# Updates
Hasta el momento, ya tenemos un modelo con metricas decentes, el cual demostró tener buenos resultados, aún así, antes de pasar el modelo a producción y desplegarlo en alguna web, quiero agregar algunas razas (clases) comunes en Chile y además aprovechar de poner en práctica algunas nuevas prácticas (valga la redundancia) que he aprendido, aparte de intentar mejorar con una nueva arquitectura las métricas obtenidas hasta el momento. Sin más preambulo, lo que haremos será:

1.   Agregar razas nuevas: `Dalmata-Akita-Bulldog Inglés`
2.   Probar nuevos modelos pre entrenados [State of the Art](https://paperswithcode.com/task/image-classification) en [keras](https://keras.io/api/applications/) (Elegir los mejores 3) `feature extraction`, de acá se obtiene: `modelo_1, modelo_2 y modelo_3`
3.   Fine-tuning modelos 1,2 y 3 `modelo_4, modelo_5 y modelo_6`
4.   Hacer un ensemble de los modelos pre entrenados escogidos `modelo_7`
5.   Añadir en el Top una red neuronal luego del ensemble `modelo_8`
6.   Agregar Data Augmentation en el mejor modelo (Data Augmentation dentro del modelo) `modelo_9`

Recordar: Utilizar nuevos callbacks aprendidos

# Librerias

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import tensorflow as tf
import tensorflow_hub as hub
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing
```

# Helper Functions

```python
# Plot the validation and training data separately
def plot_loss_curves(history):
  """
  Returns separate loss curves for training and validation metrics.
  """
  loss = history.history['loss']
  val_loss = history.history['val_loss']

  accuracy = history.history['accuracy']
  val_accuracy = history.history['val_accuracy']

  epochs = range(len(history.history['loss']))

  # Plot loss
  plt.plot(epochs, loss, label='training_loss')
  plt.plot(epochs, val_loss, label='val_loss')
  plt.title('Loss')
  plt.xlabel('Epochs')
  plt.legend()

  # Plot accuracy
  plt.figure()
  plt.plot(epochs, accuracy, label='training_accuracy')
  plt.plot(epochs, val_accuracy, label='val_accuracy')
  plt.title('Accuracy')
  plt.xlabel('Epochs')
  plt.legend();

# Convierte una carpeta en formato zip, esto para que pueda ser descargada en google colab
def zip_folder(folder_dir):
  import shutil
  shutil.make_archive(folder_dir, 'zip', folder_dir)
```

# Callbacks

```python
import datetime
import os

def create_tensorboard_callback(dir_name, experiment_name):
  # Si no esta el directorio dir_name creado, lo creamos
  if dir_name not in os.listdir():
    os.mkdir(dir_name)

  log_dir = dir_name + "/" + experiment_name + "/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
  tensorboard_callback = tf.keras.callbacks.TensorBoard(
      log_dir=log_dir
  )
  print(f"Saving TensorBoard log files to: {log_dir}")
  return tensorboard_callback
```

```python

```

# Data Directories

```python
model_image_size_map = {
  "efficientnet_b0": 224,
  "efficientnet_b1": 240,
  "efficientnet_b2": 260,
  "efficientnet_b3": 300,
  "efficientnet_b4": 380,
  "efficientnet_b5": 456,
  "efficientnet_b6": 528,
  "efficientnet_b7": 600,
  "inception_v3": 299,
  "inception_resnet_v2": 299,
  "nasnet_large": 331,
  "pnasnet_large": 331,
}

```

```python
# Ahora se encuentran las razas agregadas
train_dir = '/content/drive/MyDrive/Colab Data/Proyecto buscomiperro/train'
test_dir = '/content/drive/MyDrive/Colab Data/Proyecto buscomiperro/test'

```

# Diferencias TF hub vs Keras Applications

Si bien trabajan sobre los mismos modelos pre entrenados, hay grandes diferencias en cuanto a las condiciones de entrada (Shape) y sus respectivas salidas. Estas son algunas diferencias:

- TF Hub version returns logits (before the softmax function that turns them into probabilities), which is a common practice, because the cross-entropy loss can be computed with greater numerical stability from the logits.
- The TF Hub model assumes float32 inputs in the range of [0,1], which is what you get from tf.image.decode_jpeg(...) followed by tf.image.convert_image_dtype(..., tf.float32). The Keras code uses a model-specific range (likely [-1,+1]).
- The TF Hub model reflects the original SLIM checkpoint more completely in returning all its 1001 output classes. As stated in the ImageNetLabels.txt linked from the documentation, the added class 0 is "background" (aka. "stuff"). That is what object detection uses to indicate image background as opposed to an object of any known class.

Fuente : https://stackoverflow.com/questions/60251715/difference-between-keras-and-tensorflow-hub-version-of-mobilenetv2

# Pretrained Models
Cargamos los modelos pre entrenados usando la API de Keras o Tensorflow Hub.

Hay que tener mucho ojo en el preprocesamiento de las imagenes para cada modelo pre entrenado

```python
# Vamos a crear una función para poder definir cada modelo pre entrenado
def create_model(model_url, img_size, scale = True, trainable = False):

  # Data augmentation / Scaling
  data_augmentation = tf.keras.Sequential([
                                        #preprocessing.RandomFlip("horizontal"),
                                        #preprocessing.RandomRotation(0.2),
                                        #preprocessing.RandomZoom(0.2),
                                        #preprocessing.RandomHeight(0.2),
                                        #preprocessing.RandomWidth(0.2),
                                        preprocessing.Rescaling(1./255) # keep for ResNet50V2, remove for EfficientNetB0
                                        ], name ="data_augmentation")

  feature_extractor_layer = hub.KerasLayer(model_url,
                                           trainable= trainable ,
                                           name='feature_extraction_layer',
                                           input_shape= img_size+(3,)) # define the input image shape

  model = tf.keras.Sequential([
    layers.InputLayer(input_shape= img_size + (3,)),
    data_augmentation,
    feature_extractor_layer, # use the feature extraction layer as the base
    layers.Dense(NUM_CLASSES, activation='softmax', name='output_layer') # create our own output layer
  ])

  return model
```

```python
def create_model_application(model_application, preprocess_input, img_size, model_name = 'Model'):
  from tensorflow.keras.utils import plot_model

  base_model = model_application(include_top = False)
  base_model.trainable = False

  plot_model(base_model, to_file = model_name + '_plot.png')

  # Arquitectura

  inputs = tf.keras.layers.Input(img_size + (3,), name = 'Input_Layer')

  x = preprocess_input(inputs)

  x = base_model(x, training = False) # se me olvido poner trainable = False D: no me la container amarillo

  x = tf.keras.layers.GlobalAveragePooling2D(name = 'global_average_layer')(x)

  outputs = tf.keras.layers.Dense(NUM_CLASSES, activation='softmax', name = 'Output_Layer')(x)

  model = tf.keras.Model(inputs, outputs, name = model_name)

  model.compile(loss='categorical_crossentropy',
              optimizer=tf.keras.optimizers.Adam(),
              metrics=["accuracy"])

  return model

```

## InceptionV3 - TF hub

| Model       | Size  | Top-1 Accuracy | Top-5 Accuracy | Parameters | Depth |
|-------------|-------|----------------|----------------|------------|-------|
| InceptionV3 | 92 MB | 0.779          | 0.937          | 23,851,784 | 159   |

Debido a que esta fue la arquitectura que probamos anteriormente, comenzaremos con InceptionV3, viendo en TF Hub me di cuenta que existian ademas pesos pre entrenados con un dataset llamado iNaturalist, y ya que nuestro proyecto se enfoca a animales, probaré también estos pesos para ver que tal.

*   [Inception V3- ImageNet](https://tfhub.dev/google/imagenet/inception_v3/feature_vector/5)
*   [Inception V3- iNaturalist](https://tfhub.dev/google/inaturalist/inception_v3/feature_vector/5)

**Input Shape Requeriments**

The output is a batch of feature vectors. For each input image, the feature vector has size num_features = 2048. The feature vectors can then be used further, e.g., for classification as above.

The input images are expected to have color values in the range [0,1], following the common image input conventions. The expected size of the input images is **height x width = 299 x 299** pixels by default, but other input sizes are possible (within limits).

```python
IMG_SIZE = (299, 299)
BATCH_SIZE = 32
NUM_CLASSES = 125 # Las 122 anteriores + las 3 nuevas.
```

125 clases ya que se agregaron 3 más

DATA

```python
train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                 label_mode='categorical',
                                                                 image_size = IMG_SIZE,
                                                                 batch_size = BATCH_SIZE
                                                                 )

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode = 'categorical',
                                                                image_size = IMG_SIZE,
                                                                batch_size = BATCH_SIZE
                                                                )

```

    Found 8594 files belonging to 125 classes.
    Found 2108 files belonging to 125 classes.

```python
# Feature vectors of images with Inception V3 trained on ImageNet (ILSVRC-2012-CLS).
url_IncImgNet = 'https://tfhub.dev/google/imagenet/inception_v3/feature_vector/5'

# Feature vectors of images with Inception V3 trained on the iNaturalist (iNat) 2017 dataset.
url_IncNaturalist = 'https://tfhub.dev/google/inaturalist/inception_v3/feature_vector/5'
```

### ImageNet

```python
IncImgNet_model = create_model(url_IncImgNet , IMG_SIZE, scale = True) # Inception necesita las imagenes en un rango entre [0,1]
```

```python
IncImgNet_model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    data_augmentation (Sequentia (None, 299, 299, 3)       0
    _________________________________________________________________
    feature_extraction_layer (Ke (None, 2048)              21802784
    _________________________________________________________________
    output_layer (Dense)         (None, 125)               256125
    =================================================================
    Total params: 22,058,909
    Trainable params: 256,125
    Non-trainable params: 21,802,784
    _________________________________________________________________

Es importante entender los shape de estas etapas, según la documentación, de InceptionV3 sale (luego del respectivo Flatten) un vector de tamaño 2048. Luego viene un Dense layer de 125 nodos con función de activación Softmax ya que tenemos 125 clases. Es por esto que los 2048 estan conectados a los 125 nodos de la última layer, además cada nodo tiene un parámetro de Bias, resultando en:

\begin{equation}
parametros = 2048*125 + 125 = 256125
\end{equation}

```python
# Compilamos
IncImgNet_model.compile(loss='categorical_crossentropy',
                     optimizer=tf.keras.optimizers.Adam(),
                     metrics=['accuracy'])
```

```python
# Fit the model
IncImgNet_history = IncImgNet_model.fit(train_data,
                                        epochs=5,
                                        steps_per_epoch=len(train_data),
                                        validation_data=test_data,
                                        validation_steps=len(test_data),
                                        # Add TensorBoard callback to model (callbacks parameter takes a list)
                                        callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                              experiment_name="InceptionV3-ImageNet")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/InceptionV3-ImageNet/20210526-193507
    Epoch 1/5
    269/269 [==============================] - 1268s 4s/step - loss: 1.1313 - accuracy: 0.7767 - val_loss: 0.4360 - val_accuracy: 0.8743
    Epoch 2/5
    269/269 [==============================] - 72s 265ms/step - loss: 0.2790 - accuracy: 0.9181 - val_loss: 0.3922 - val_accuracy: 0.8757
    Epoch 3/5
    269/269 [==============================] - 72s 265ms/step - loss: 0.1908 - accuracy: 0.9446 - val_loss: 0.3795 - val_accuracy: 0.8824
    Epoch 4/5
    269/269 [==============================] - 72s 264ms/step - loss: 0.1413 - accuracy: 0.9607 - val_loss: 0.3779 - val_accuracy: 0.8771
    Epoch 5/5
    269/269 [==============================] - 72s 265ms/step - loss: 0.1103 - accuracy: 0.9716 - val_loss: 0.3687 - val_accuracy: 0.8781

```python
plot_loss_curves(IncImgNet_history)
```

```python
IncImgNet_model.save('IncImgNet.h5')
```

### iNaturalist

```python
IncNaturalist_model = create_model(url_IncNaturalist, IMG_SIZE, scale = True, trainable = False)
```

```python
IncNaturalist_model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    data_augmentation (Sequentia (None, 299, 299, 3)       0
    _________________________________________________________________
    feature_extraction_layer (Ke (None, 2048)              21802784
    _________________________________________________________________
    output_layer (Dense)         (None, 125)               256125
    =================================================================
    Total params: 22,058,909
    Trainable params: 256,125
    Non-trainable params: 21,802,784
    _________________________________________________________________

```python
IncNaturalist_model.compile(loss='categorical_crossentropy',
                     optimizer=tf.keras.optimizers.Adam(),
                     metrics=['accuracy'])
```

```python
IncNaturalist_model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    data_augmentation (Sequentia (None, 299, 299, 3)       0
    _________________________________________________________________
    feature_extraction_layer (Ke (None, 2048)              21802784
    _________________________________________________________________
    output_layer (Dense)         (None, 125)               256125
    =================================================================
    Total params: 22,058,909
    Trainable params: 256,125
    Non-trainable params: 21,802,784
    _________________________________________________________________

```python
# Fit the model
IncNaturalist_history = IncNaturalist_model.fit(train_data,
                                        epochs=5,
                                        steps_per_epoch=len(train_data),
                                        validation_data=test_data,
                                        validation_steps=len(test_data),
                                        # Add TensorBoard callback to model (callbacks parameter takes a list)
                                        callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                              experiment_name="InceptionV3-iNaturalist")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/InceptionV3-iNaturalist/20210527-043754
    Epoch 1/5
    269/269 [==============================] - 73s 271ms/step - loss: 0.9931 - accuracy: 0.7647 - val_loss: 2.0648 - val_accuracy: 0.4540
    Epoch 2/5
    269/269 [==============================] - 72s 266ms/step - loss: 0.8584 - accuracy: 0.8093 - val_loss: 2.0329 - val_accuracy: 0.4711
    Epoch 3/5
    269/269 [==============================] - 72s 267ms/step - loss: 0.7452 - accuracy: 0.8414 - val_loss: 2.0623 - val_accuracy: 0.4639
    Epoch 4/5
    269/269 [==============================] - 72s 265ms/step - loss: 0.6546 - accuracy: 0.8671 - val_loss: 2.0757 - val_accuracy: 0.4696
    Epoch 5/5
    269/269 [==============================] - 72s 266ms/step - loss: 0.5749 - accuracy: 0.8949 - val_loss: 2.0794 - val_accuracy: 0.4720

```python
plot_loss_curves(IncNaturalist_history)
```

```python

```

### Conclusión
En base a los resultados obtenidos, podemos definir el `modelo 1 = IncImgNet`.

```python

```

## InceptionV3 - Keras Applications

Este proceso solo lo hago con fines de aprendizaje, ya que no deberian variar mucho los resultados, pero creo que es bueno poder prácticar ambas herramientas.

De todas formas, una diferencia inmediata que pude encontrar es que utilizando la API de Keras, podemos obtener el preprocesamiento para cada modelo pre entrenado.

Incluso dentro de este enfoque, hay otros 2 enfoques distintos, Sequential API vs Functional API. En el pasado trabajamos con el enfoque Secuencial. La principal diferencia entre estas APIS es que la API funcional es más flexible que la secuencial, ya que esta ultima te permite crear modelos layer por layer, y **no** permite: Share Layers, Tener branches, Multiples entradas ni multiple salidas. En cambio con la API funcional si se puede. [Más Info](https://hanifi.medium.com/sequential-api-vs-functional-api-model-in-keras-266823d7cd5e).
Por razones de estudio, utilizaré la api funcional.

**Input Shape Requeriments**
Es importante destacar que, si bien en TF Hub se esperaba un rango de entrada de [0,1], en Keras Applications se espera que sea `[-1,1]`. Esto es una gran diferencia, pero para esto nos sirve la función `preprocess_input` que cada modelo pre entrenado tiene consigo. Los pixeles se mantienen, siendo estos `299x299`

```python
from tensorflow.keras.applications import InceptionV3

# modelo pre entrenado
base_model = InceptionV3(include_top = False, input_shape= IMG_SIZE + (3,)) # Pondremos nuestro propio Top Layer (Dense(125))

# Congelamos el modelo para mantener los patrones pre entrenados
base_model.trainable = False

```

    Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/inception_v3/inception_v3_weights_tf_dim_ordering_tf_kernels.h5
    96116736/96112376 [==============================] - 1s 0us/step

```python
# Veamos la impresionante arquitectura de este modelo
from tensorflow.keras.utils import plot_model
plot_model(base_model, show_shapes = True)
```

SHEEEEEEESH

```python
from tensorflow.keras.applications.inception_v3 import preprocess_input # Preprocesa las imaganes a lo que acomoda el modelo pre entrenado

# Definimos la Arquitectura con la API Funcional

inputs = tf.keras.layers.Input(IMG_SIZE + (3,), name = 'Input_Layer') # Debe haber un input layer ya que el enfoque funcional lo requiere

x = preprocess_input(inputs)

x = base_model(x)

# A diferencia de Tensorflow HUB, la salida del modelo base no es plana, para poder hacerla plana, una opción es aplicar un Average Pool,
# esto tambien se hace en el Top por Defecto de Inception (para comprobarlo puedes cargar el modelo base con include_top = True y echar un vistazo
# a los ultimos layers)
x = tf.keras.layers.GlobalAveragePooling2D(name="global_average_pooling_layer")(x)

outputs = tf.keras.layers.Dense(NUM_CLASSES, activation='softmax', name = 'Output_layer')(x)

IncKA_model = tf.keras.Model(inputs, outputs) # Se construye el modelo

```

```python
IncKA_model.summary()
```

    Model: "model_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    Input_Layer (InputLayer)     [(None, 299, 299, 3)]     0
    _________________________________________________________________
    tf.math.truediv_1 (TFOpLambd (None, 299, 299, 3)       0
    _________________________________________________________________
    tf.math.subtract_1 (TFOpLamb (None, 299, 299, 3)       0
    _________________________________________________________________
    inception_v3 (Functional)    (None, None, None, 2048)  21802784
    _________________________________________________________________
    global_average_pooling_layer (None, 2048)              0
    _________________________________________________________________
    Output_layer (Dense)         (None, 125)               256125
    =================================================================
    Total params: 22,058,909
    Trainable params: 256,125
    Non-trainable params: 21,802,784
    _________________________________________________________________

Veamos que los parametros son exactamente igual a los que teniamos anteriormente

```python
# Compilamos
IncKA_model.compile(loss='categorical_crossentropy',
              optimizer=tf.keras.optimizers.Adam(),
              metrics=["accuracy"])
```

```python
# Fit
IncKA_history = IncKA_model.fit(train_data,
                                epochs=5,
                                steps_per_epoch=len(train_data),
                                validation_data=test_data,
                                validation_steps=len(test_data),
                                # Add TensorBoard callback to model (callbacks parameter takes a list)
                                callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                      experiment_name="InceptionV3-FunctionalAPI")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/InceptionV3-FunctionalAPI/20210527-215004
    Epoch 1/5
    269/269 [==============================] - 2007s 7s/step - loss: 1.1663 - accuracy: 0.7704 - val_loss: 0.4418 - val_accuracy: 0.8695
    Epoch 2/5
    269/269 [==============================] - 74s 273ms/step - loss: 0.2858 - accuracy: 0.9149 - val_loss: 0.3935 - val_accuracy: 0.8748
    Epoch 3/5
    269/269 [==============================] - 74s 272ms/step - loss: 0.1961 - accuracy: 0.9440 - val_loss: 0.3923 - val_accuracy: 0.8809
    Epoch 4/5
    269/269 [==============================] - 74s 272ms/step - loss: 0.1440 - accuracy: 0.9601 - val_loss: 0.3767 - val_accuracy: 0.8743
    Epoch 5/5
    269/269 [==============================] - 74s 272ms/step - loss: 0.1101 - accuracy: 0.9716 - val_loss: 0.3670 - val_accuracy: 0.8876

```python
plot_loss_curves(IncKA_history)
```

Resultados bastantes similares, yo creo que la diferencia radica en la aleatoridad inicial de los pesos de las Top Layers.

```python
IncKA_model.save('IncImgNet-FunAPI.h5')
```

## Xception

| Model    | Size  | Top-1 Accuracy | Top-5 Accuracy | Parameters | Depth |
|----------|-------|----------------|----------------|------------|-------|
| Xception | 88 MB | 0.790          | 0.945          | 22,910,480 | 126   |

> TFhub no tiene la aquitectura Xception incorporada, por lo que recurriremos a la APi de Keras

https://keras.io/api/applications/xception/

```python
IMG_SIZE = (299, 299)
BATCH_SIZE = 32
NUM_CLASSES = 125 # Las 122 anteriores + las 3 nuevas.
```

DATA

```python
train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                 label_mode='categorical',
                                                                 image_size = IMG_SIZE,
                                                                 batch_size = BATCH_SIZE
                                                                 )

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode = 'categorical',
                                                                image_size = IMG_SIZE,
                                                                batch_size = BATCH_SIZE
                                                                )

```

    Found 8594 files belonging to 125 classes.
    Found 2108 files belonging to 125 classes.

### Xception - Keras Applications

**Input Shape Requeriments**
- 299x299
- range [-1,1] (`preprocess_input`)

```python
from tensorflow.keras.applications import Xception
from tensorflow.keras.applications.xception import preprocess_input

Xception_model = create_model_application(Xception, preprocess_input, IMG_SIZE)
```

    Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/xception/xception_weights_tf_dim_ordering_tf_kernels_notop.h5
    83689472/83683744 [==============================] - 1s 0us/step

```python
Xception_model.summary()
```

    Model: "model"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    Input_Layer (InputLayer)     [(None, 299, 299, 3)]     0
    _________________________________________________________________
    tf.math.truediv (TFOpLambda) (None, 299, 299, 3)       0
    _________________________________________________________________
    tf.math.subtract (TFOpLambda (None, 299, 299, 3)       0
    _________________________________________________________________
    xception (Functional)        (None, None, None, 2048)  20861480
    _________________________________________________________________
    global_average_layer (Global (None, 2048)              0
    _________________________________________________________________
    Output_Layer (Dense)         (None, 125)               256125
    =================================================================
    Total params: 21,117,605
    Trainable params: 256,125
    Non-trainable params: 20,861,480
    _________________________________________________________________

```python
# Fit
Xception_history = Xception_model.fit(train_data,
                                epochs=5,
                                steps_per_epoch=len(train_data),
                                validation_data=test_data,
                                validation_steps=len(test_data),
                                # Add TensorBoard callback to model (callbacks parameter takes a list)
                                callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                      experiment_name="Xception")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/Xception/20210528-165207
    Epoch 1/5

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

    269/269 [==============================] - 2073s 7s/step - loss: 1.1696 - accuracy: 0.7974 - val_loss: 0.4182 - val_accuracy: 0.8971
    Epoch 2/5
    269/269 [==============================] - 122s 452ms/step - loss: 0.2905 - accuracy: 0.9246 - val_loss: 0.3509 - val_accuracy: 0.8999
    Epoch 3/5
    269/269 [==============================] - 123s 454ms/step - loss: 0.2143 - accuracy: 0.9416 - val_loss: 0.3342 - val_accuracy: 0.9004
    Epoch 4/5
    269/269 [==============================] - 122s 453ms/step - loss: 0.1697 - accuracy: 0.9543 - val_loss: 0.3281 - val_accuracy: 0.8980
    Epoch 5/5
    269/269 [==============================] - 122s 453ms/step - loss: 0.1402 - accuracy: 0.9629 - val_loss: 0.3208 - val_accuracy: 0.9004

Aparentemente tiene mejores resultados ($\approx +3\%$) que InceptionV3

```python
Xception_model.evaluate(test_data)
```

    66/66 [==============================] - 23s 335ms/step - loss: 0.3208 - accuracy: 0.9004

    [0.3207794427871704, 0.900379478931427]

```python
plot_loss_curves(Xception_history)
```

```python
Xception_model.save('Xception.h5')
```

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

```python
zip_folder('tensorflow_hub')
```

## ResNet152V2

| Model       | Size   | Top-1 Accuracy | Top-5 Accuracy | Parameters | Depth |
|-------------|--------|----------------|----------------|------------|-------|
| ResNet152V2 | 232 MB | 0.780          | 0.942          | 60,380,648 |       |

```python
IMG_SIZE = (224, 224)
BATCH_SIZE = 32
NUM_CLASSES = 125 # Las 122 anteriores + las 3 nuevas.
```

```python
train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                 label_mode='categorical',
                                                                 image_size = IMG_SIZE,
                                                                 batch_size = BATCH_SIZE
                                                                 )

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode = 'categorical',
                                                                image_size = IMG_SIZE,
                                                                batch_size = BATCH_SIZE
                                                                )
```

    Found 8594 files belonging to 125 classes.
    Found 2108 files belonging to 125 classes.

### TF HUB

**Shape Requeriments**
The input images are expected to have color values in the range `[0,1]`, following the common image input conventions. The expected size of the input images is height x width = `224 x 224` pixels by default, but other input sizes are possible (within limits).

https://tfhub.dev/google/imagenet/resnet_v2_152/feature_vector/5

```python
url_resnet = 'https://tfhub.dev/google/imagenet/resnet_v2_152/feature_vector/5'
```

```python
resnet_hub_model = create_model(url_resnet, IMG_SIZE, scale = True, trainable = False)
```

```python
resnet_hub_model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    data_augmentation (Sequentia (None, 224, 224, 3)       0
    _________________________________________________________________
    feature_extraction_layer (Ke (None, 2048)              58331648
    _________________________________________________________________
    output_layer (Dense)         (None, 125)               256125
    =================================================================
    Total params: 58,587,773
    Trainable params: 256,125
    Non-trainable params: 58,331,648
    _________________________________________________________________

```python
# Compilamos
resnet_hub_model.compile(loss='categorical_crossentropy',
                     optimizer=tf.keras.optimizers.Adam(),
                     metrics=['accuracy'])
```

```python
# Fit the model
resnet_hub_history = resnet_hub_model.fit(train_data,
                                          epochs=5,
                                          steps_per_epoch=len(train_data),
                                          validation_data=test_data,
                                          validation_steps=len(test_data),
                                          # Add TensorBoard callback to model (callbacks parameter takes a list)
                                          callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                                experiment_name="resnet_hub")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/resnet_hub/20210529-042304
    Epoch 1/5
    269/269 [==============================] - 1280s 5s/step - loss: 1.4942 - accuracy: 0.6397 - val_loss: 0.8287 - val_accuracy: 0.7652
    Epoch 2/5
    269/269 [==============================] - 99s 367ms/step - loss: 0.4356 - accuracy: 0.8684 - val_loss: 0.8413 - val_accuracy: 0.7623
    Epoch 3/5
    269/269 [==============================] - 98s 364ms/step - loss: 0.2267 - accuracy: 0.9405 - val_loss: 0.8091 - val_accuracy: 0.7690
    Epoch 4/5
    269/269 [==============================] - 98s 363ms/step - loss: 0.1292 - accuracy: 0.9720 - val_loss: 0.8343 - val_accuracy: 0.7694
    Epoch 5/5
    269/269 [==============================] - 98s 363ms/step - loss: 0.0826 - accuracy: 0.9862 - val_loss: 0.8382 - val_accuracy: 0.7676

```python
zip_folder('tensorflow_hub')
```

### Keras Application

https://keras.io/api/applications/resnet/#resnet152v2-function

```python
from tensorflow.keras.applications import ResNet152V2
from tensorflow.keras.applications.resnet_v2 import preprocess_input

base_model = ResNet152V2

resnet_ka_model = create_model_application(base_model, preprocess_input, IMG_SIZE, model_name = 'resnet')
```

    Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet152v2_weights_tf_dim_ordering_tf_kernels_notop.h5
    234553344/234545216 [==============================] - 3s 0us/step
    dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.604814 to fit

```python
resnet_ka_model.summary()
```

    Model: "resnet"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    Input_Layer (InputLayer)     [(None, 224, 224, 3)]     0
    _________________________________________________________________
    tf.math.truediv (TFOpLambda) (None, 224, 224, 3)       0
    _________________________________________________________________
    tf.math.subtract (TFOpLambda (None, 224, 224, 3)       0
    _________________________________________________________________
    resnet152v2 (Functional)     (None, None, None, 2048)  58331648
    _________________________________________________________________
    global_average_layer (Global (None, 2048)              0
    _________________________________________________________________
    Output_Layer (Dense)         (None, 125)               256125
    =================================================================
    Total params: 58,587,773
    Trainable params: 256,125
    Non-trainable params: 58,331,648
    _________________________________________________________________

```python
# Fit
resnet_ka_history = resnet_ka_model.fit(train_data,
                                epochs=5,
                                steps_per_epoch=len(train_data),
                                validation_data=test_data,
                                validation_steps=len(test_data),
                                # Add TensorBoard callback to model (callbacks parameter takes a list)
                                callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                      experiment_name="resnet_KA")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/resnet_KA/20210529-143027

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

    Epoch 1/5
    269/269 [==============================] - 1297s 5s/step - loss: 1.5197 - accuracy: 0.6317 - val_loss: 0.8952 - val_accuracy: 0.7410
    Epoch 2/5
    269/269 [==============================] - 100s 369ms/step - loss: 0.4386 - accuracy: 0.8663 - val_loss: 0.8493 - val_accuracy: 0.7595
    Epoch 3/5
    269/269 [==============================] - 101s 374ms/step - loss: 0.2328 - accuracy: 0.9360 - val_loss: 0.8525 - val_accuracy: 0.7576
    Epoch 4/5
    269/269 [==============================] - 101s 374ms/step - loss: 0.1328 - accuracy: 0.9713 - val_loss: 0.8550 - val_accuracy: 0.7704
    Epoch 5/5
    269/269 [==============================] - 102s 379ms/step - loss: 0.0825 - accuracy: 0.9869 - val_loss: 0.8418 - val_accuracy: 0.7728

Vemos que Resnet consigue unos resultados buenos, pero no tanto como los otros estimadores, aún asi, ya que queremos escoger 3 modelos, este quedaria seleccionado.

```python
resnet_ka_model.evaluate(test_data)
```

    66/66 [==============================] - 20s 298ms/step - loss: 0.8418 - accuracy: 0.7728

    [0.8417608737945557, 0.7727704048156738]

```python
zip_folder('tensorflow_hub')
```

## EfficientNet

### EfficientNet 7

| Model          | Size   | Top-1 Accuracy | Top-5 Accuracy | Parameters | Depth |
|----------------|--------|----------------|----------------|------------|-------|
| EfficientNetB7 | 256 MB | -              | -              | 66,658,687 | -     |

```python
IMG_SIZE = (600,600)
NUM_CLASSES = 125
BATCH_SIZE = 32
```

```python
train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                 label_mode='categorical',
                                                                 image_size = IMG_SIZE,
                                                                 batch_size = BATCH_SIZE
                                                                 )

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode = 'categorical',
                                                                image_size = IMG_SIZE,
                                                                batch_size = BATCH_SIZE
                                                                )
```

    Found 8594 files belonging to 125 classes.
    Found 2108 files belonging to 125 classes.

**Tensor Hub**

> For this module, the size of the input image is flexible, but it would be best to match the model training input, which is height x width = 600 x 600 pixels for this model. The input images are expected to have color values in the range [0,1], following the common image input conventions.

Fine-tuning

https://tfhub.dev/tensorflow/efficientnet/b7/feature-vector/1

```python
url_EfficientNet7 = 'https://tfhub.dev/tensorflow/efficientnet/b7/feature-vector/1'
```

```python
EfficientNet7_hub_model = create_model(url_EfficientNet7, IMG_SIZE, scale = True, trainable = False)
```

```python
EfficientNet7_hub_model.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    data_augmentation (Sequentia (None, 600, 600, 3)       0
    _________________________________________________________________
    feature_extraction_layer (Ke (None, 2560)              64097680
    _________________________________________________________________
    output_layer (Dense)         (None, 125)               320125
    =================================================================
    Total params: 64,417,805
    Trainable params: 320,125
    Non-trainable params: 64,097,680
    _________________________________________________________________

```python
# Compilamos
EfficientNet7_hub_model.compile(loss='categorical_crossentropy',
                     optimizer=tf.keras.optimizers.Adam(),
                     metrics=['accuracy'])
```

```python
# Fit the model
EfficientNet7_hub_history = EfficientNet7_hub_model.fit(train_data,
                                          epochs=5,
                                          steps_per_epoch=len(train_data),
                                          validation_data=test_data,
                                          validation_steps=len(test_data),
                                          # Add TensorBoard callback to model (callbacks parameter takes a list)
                                          callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                                experiment_name="EfficientNet7_hub")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/EfficientNet7_hub/20210530-032803
    Epoch 1/5
    269/269 [==============================] - 2808s 10s/step - loss: 1.0858 - accuracy: 0.8593 - val_loss: 0.2259 - val_accuracy: 0.9492
    Epoch 2/5
    269/269 [==============================] - 927s 3s/step - loss: 0.1640 - accuracy: 0.9632 - val_loss: 0.1720 - val_accuracy: 0.9535
    Epoch 3/5
    269/269 [==============================] - 927s 3s/step - loss: 0.1111 - accuracy: 0.9737 - val_loss: 0.1613 - val_accuracy: 0.9554
    Epoch 4/5
    269/269 [==============================] - 927s 3s/step - loss: 0.0833 - accuracy: 0.9800 - val_loss: 0.1584 - val_accuracy: 0.9549
    Epoch 5/5
    269/269 [==============================] - 927s 3s/step - loss: 0.0657 - accuracy: 0.9831 - val_loss: 0.1558 - val_accuracy: 0.9568

```python
EfficientNet7_hub_model.evaluate(test_data)
```

    66/66 [==============================] - 183s 3s/step - loss: 0.1558 - accuracy: 0.9568

    [0.15576839447021484, 0.9568310976028442]

```python
EfficientNet7_hub_model.save('EfficientNet7_hub.h5')
```

```python
zip_folder('tensorflow_hub')
```

El mejor resultado hasta el momento!!

**Keras Applications**

https://keras.io/api/applications/efficientnet/#efficientnetb7-function

> For EfficientNet, input preprocessing is included as part of the model (as a Rescaling layer), and thus tf.keras.applications.efficientnet.preprocess_input is actually a pass-through function. EfficientNet models expect their inputs to be float tensors of pixels with values in the [0-255] range.

```python
from tensorflow.keras.applications import EfficientNetB7
from tensorflow.keras.applications.efficientnet import preprocess_input

EfficientNet7_ka_model = create_model_application(EfficientNetB7, preprocess_input, IMG_SIZE, model_name = 'EfficientNet7')
```

    Downloading data from https://storage.googleapis.com/keras-applications/efficientnetb7_notop.h5
    258080768/258076736 [==============================] - 2s 0us/step
    dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.414274 to fit

```python
EfficientNet7_ka_model.summary()
```

    Model: "EfficientNet7"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    Input_Layer (InputLayer)     [(None, 600, 600, 3)]     0
    _________________________________________________________________
    efficientnetb7 (Functional)  (None, None, None, 2560)  64097687
    _________________________________________________________________
    global_average_layer (Global (None, 2560)              0
    _________________________________________________________________
    Output_Layer (Dense)         (None, 125)               320125
    =================================================================
    Total params: 64,417,812
    Trainable params: 320,125
    Non-trainable params: 64,097,687
    _________________________________________________________________

```python
# Fit
EfficientNet7_ka_history = EfficientNet7_ka_model.fit(train_data,
                                epochs=5,
                                steps_per_epoch=len(train_data),
                                validation_data=test_data,
                                validation_steps=len(test_data),
                                # Add TensorBoard callback to model (callbacks parameter takes a list)
                                callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                      experiment_name="EfficientNet_KA")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/EfficientNet_KA/20210530-183201

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

    Epoch 1/5
    269/269 [==============================] - 2111s 8s/step - loss: 1.1452 - accuracy: 0.8480 - val_loss: 0.2516 - val_accuracy: 0.9407
    Epoch 2/5
    269/269 [==============================] - 952s 4s/step - loss: 0.1995 - accuracy: 0.9547 - val_loss: 0.1883 - val_accuracy: 0.9483
    Epoch 3/5
    269/269 [==============================] - 951s 4s/step - loss: 0.1407 - accuracy: 0.9637 - val_loss: 0.1737 - val_accuracy: 0.9492
    Epoch 4/5
    269/269 [==============================] - 951s 4s/step - loss: 0.1095 - accuracy: 0.9724 - val_loss: 0.1681 - val_accuracy: 0.9521
    Epoch 5/5
    269/269 [==============================] - 952s 4s/step - loss: 0.0896 - accuracy: 0.9761 - val_loss: 0.1649 - val_accuracy: 0.9554

```python
EfficientNet7_ka_model.evaluate(test_data)
```

    66/66 [==============================] - 185s 3s/step - loss: 0.1649 - accuracy: 0.9554

    [0.16494086384773254, 0.955407977104187]

```python
EfficientNet7_ka_model.save('EfficientNet7_ka.h5')
```

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

```python
zip_folder('/content/tensorflow_hub')
```

```python

```

## InceptionResNetV2

> Desde ahora solo ocupare un enfoque (TFHUB o Applications)

| Model             | Size   | Top-1 Accuracy | Top-5 Accuracy | Parameters | Depth |
|-------------------|--------|----------------|----------------|------------|-------|
| InceptionResNetV2 | 215 MB | 0.803          | 0.953          | 55,873,736 | 572   |

**Shape Requeriments**

- `img_dim = 299x299`
- `range_pixel = [-1,1]`

Data

```python
IMG_SIZE = (299,299)
NUM_CLASSES = 125
BATCH_SIZE = 32
```

```python
train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                 label_mode='categorical',
                                                                 image_size = IMG_SIZE,
                                                                 batch_size = BATCH_SIZE
                                                                 )

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode = 'categorical',
                                                                image_size = IMG_SIZE,
                                                                batch_size = BATCH_SIZE
                                                                )
```

    Found 8594 files belonging to 125 classes.
    Found 2108 files belonging to 125 classes.

```python
from tensorflow.keras.applications import InceptionResNetV2
from tensorflow.keras.applications.inception_resnet_v2 import preprocess_input

InceptionResNetV2_model = create_model_application(InceptionResNetV2, preprocess_input, IMG_SIZE, model_name = 'InceptionResNetV2')
```

    dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.591003 to fit

```python
InceptionResNetV2_model.summary()
```

    Model: "InceptionResNetV2"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    Input_Layer (InputLayer)     [(None, 299, 299, 3)]     0
    _________________________________________________________________
    tf.math.truediv_2 (TFOpLambd (None, 299, 299, 3)       0
    _________________________________________________________________
    tf.math.subtract_2 (TFOpLamb (None, 299, 299, 3)       0
    _________________________________________________________________
    inception_resnet_v2 (Functio (None, None, None, 1536)  54336736
    _________________________________________________________________
    global_average_layer (Global (None, 1536)              0
    _________________________________________________________________
    Output_Layer (Dense)         (None, 125)               192125
    =================================================================
    Total params: 54,528,861
    Trainable params: 192,125
    Non-trainable params: 54,336,736
    _________________________________________________________________

```python
# Fit
InceptionResNetV2_history = InceptionResNetV2_model.fit(train_data,
                                epochs=5,
                                steps_per_epoch=len(train_data),
                                validation_data=test_data,
                                validation_steps=len(test_data),
                                # Add TensorBoard callback to model (callbacks parameter takes a list)
                                callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                      experiment_name="InceptionResNetV2")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/InceptionResNetV2/20210606-025726
    Epoch 1/5
    269/269 [==============================] - 1752s 6s/step - loss: 0.9546 - accuracy: 0.8290 - val_loss: 0.3354 - val_accuracy: 0.9075
    Epoch 2/5
    269/269 [==============================] - 144s 532ms/step - loss: 0.2489 - accuracy: 0.9239 - val_loss: 0.3158 - val_accuracy: 0.9046
    Epoch 3/5
    269/269 [==============================] - 144s 534ms/step - loss: 0.2040 - accuracy: 0.9359 - val_loss: 0.3111 - val_accuracy: 0.9108
    Epoch 4/5
    269/269 [==============================] - 144s 533ms/step - loss: 0.1719 - accuracy: 0.9450 - val_loss: 0.3105 - val_accuracy: 0.9137
    Epoch 5/5
    269/269 [==============================] - 144s 533ms/step - loss: 0.1535 - accuracy: 0.9514 - val_loss: 0.3244 - val_accuracy: 0.9075

```python
InceptionResNetV2_model.evaluate(test_data)
```

    66/66 [==============================] - 29s 426ms/step - loss: 0.3244 - accuracy: 0.9075

```python
InceptionResNetV2_model.save('InceptionResNetV2.h5')
```

```python
zip_folder('/content/tensorflow_hub')
```

## NasNetLarge

https://keras.io/api/applications/nasnet/#nasnetlarge-function

| Model             | Size   | Top-1 Accuracy | Top-5 Accuracy | Parameters | Depth |
|-------------------|--------|----------------|----------------|------------|-------|
| InceptionResNetV2 | 215 MB | 0.803          | 0.953          | 55,873,736 | 572   |

**Input Shape Requeriments**

- `img_dim = 331x331`

```python
IMG_SIZE = (331,331)
NUM_CLASSES = 125
BATCH_SIZE = 32
```

```python
train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                 label_mode='categorical',
                                                                 image_size = IMG_SIZE,
                                                                 batch_size = BATCH_SIZE
                                                                 )

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode = 'categorical',
                                                                image_size = IMG_SIZE,
                                                                batch_size = BATCH_SIZE
                                                                )
```

    Found 8594 files belonging to 125 classes.
    Found 2108 files belonging to 125 classes.

```python
from tensorflow.keras.applications import NASNetLarge
from tensorflow.keras.applications.nasnet import preprocess_input

NasNetLarge_model = create_model_application(NASNetLarge, preprocess_input, IMG_SIZE, model_name = 'NasNetLarge')
```

    Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/nasnet/NASNet-large-no-top.h5
    343613440/343610240 [==============================] - 3s 0us/step

```python
NasNetLarge_model.summary()
```

    Model: "NasNetLarge"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    Input_Layer (InputLayer)     [(None, 331, 331, 3)]     0
    _________________________________________________________________
    tf.math.truediv_1 (TFOpLambd (None, 331, 331, 3)       0
    _________________________________________________________________
    tf.math.subtract_1 (TFOpLamb (None, 331, 331, 3)       0
    _________________________________________________________________
    NASNet (Functional)          (None, 11, 11, 4032)      84916818
    _________________________________________________________________
    global_average_layer (Global (None, 4032)              0
    _________________________________________________________________
    Output_Layer (Dense)         (None, 125)               504125
    =================================================================
    Total params: 85,420,943
    Trainable params: 504,125
    Non-trainable params: 84,916,818
    _________________________________________________________________

```python
# Fit
NasNetLarge_history = NasNetLarge_model.fit(train_data,
                                epochs=5,
                                steps_per_epoch=len(train_data),
                                validation_data=test_data,
                                validation_steps=len(test_data),
                                # Add TensorBoard callback to model (callbacks parameter takes a list)
                                callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                                      experiment_name="NasNetLarge")]) # name of log files
```

    Saving TensorBoard log files to: tensorflow_hub/NasNetLarge/20210602-052658

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

    Epoch 1/5
    269/269 [==============================] - 372s 1s/step - loss: 0.6397 - accuracy: 0.8771 - val_loss: 0.2373 - val_accuracy: 0.9303
    Epoch 2/5
    269/269 [==============================] - 345s 1s/step - loss: 0.1411 - accuracy: 0.9578 - val_loss: 0.2310 - val_accuracy: 0.9355
    Epoch 3/5
    269/269 [==============================] - 345s 1s/step - loss: 0.0986 - accuracy: 0.9692 - val_loss: 0.2312 - val_accuracy: 0.9322
    Epoch 4/5
    269/269 [==============================] - 345s 1s/step - loss: 0.0754 - accuracy: 0.9754 - val_loss: 0.2365 - val_accuracy: 0.9326
    Epoch 5/5
    269/269 [==============================] - 345s 1s/step - loss: 0.0586 - accuracy: 0.9817 - val_loss: 0.2373 - val_accuracy: 0.9326

```python
NasNetLarge_model.evaluate(test_data)
```

    66/66 [==============================] - 67s 1s/step - loss: 0.2373 - accuracy: 0.9326

    [0.23726095259189606, 0.9326375722885132]

```python
NasNetLarge_model.save('NasNetLarge.h5')
```

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

```python
zip_folder('/content/tensorflow_hub')
```

```python

```

# Resumen

| Model Application | Train Loss | Train Accuracy | Val Loss | Val Accuracy |
|-------------------|------------|----------------|----------|--------------|
| EfficientNet7     | 0.0896     | 0.9761         | 0.1649   | 0.9554       |
| NasNetLarge       | 0.0586     | 0.9817         | 0.2373   | 0.9326       |
| InceptionResNetV2 | 0.1532     | 0.9496         | 0.3049   | 0.9141       |
| Xception          | 0.1402     | 0.9629         | 0.3208   | 0.9004       |
| InceptionV3       | 0.1101     | 0.9716         | 0.3670   | 0.8876       |
| ResNet152V2       | 0.0825     | 0.9869         | 0.8418   | 0.7728       |

EfficientNet consigue resultados bastante buenos por si solo, de hecho cada uno solo lo entrenamos por 5 epochs, no hicimos fine tuning, hyperparameter tuning, ni data augmentation. Lo que haré ahora será stackear los mejores 3 modelos y ver que resulta de eso

# Stack Models
Se hará un stack entre EfficientNet7, NasNetLarge y InceptionResNetV2

# Deberia ser algo asi
input = inputlayer() \\
x1 = efficientnet7()(input) \\
x2 = nasnetlarge()(input) \\
x3 = inceptionresnetv2()(input) \\
x = concatenate([x1,x2,x3]) \\
output = Dense(125)(x)

Deberiamos hacerlo con la api funcional porque tenemos que concatenar layers

```python
from tensorflow.keras.applications import NASNetLarge
from tensorflow.keras.applications.nasnet import preprocess_input as nasnet_preprocess

from tensorflow.keras.applications import InceptionResNetV2
from tensorflow.keras.applications.inception_resnet_v2 import preprocess_input as incresnet_preprocess

from tensorflow.keras.applications import EfficientNetB7
from tensorflow.keras.applications.efficientnet import preprocess_input as effnet7_preprocess

```

PARAMETROS

```python
NUM_CLASSES = 125
BATCH_SIZE = 32
IMG_SIZE = (299,299)
EFFNET7_IMG_SIZE = (600,600)
NASNET_IMG_SIZE = (331,331)
INCRESNET_IMG_SIZE = (299,299)

```

DATA

```python
train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                 label_mode='categorical',
                                                                 batch_size = BATCH_SIZE,
                                                                 )

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode = 'categorical',
                                                                batch_size = BATCH_SIZE
                                                                )
```

    Found 8594 files belonging to 125 classes.
    Found 2108 files belonging to 125 classes.

```python
# base models

effnet7_model = EfficientNetB7(include_top = False)
effnet7_model.trainable = False

nasnet_model = NASNetLarge(include_top = False)
nasnet_model.trainable = False

incresnet_model = InceptionResNetV2(include_top = False)
incresnet_model.trainable = False

input = tf.keras.layers.Input(shape = (256,256,3))

# RESIZE LAYER

input_effnet7 = tf.keras.layers.experimental.preprocessing.Resizing(EFFNET7_IMG_SIZE[0],EFFNET7_IMG_SIZE[1])(input)
input_nasnet = tf.keras.layers.experimental.preprocessing.Resizing(NASNET_IMG_SIZE[0],NASNET_IMG_SIZE[1])(input)
input_incresnet = tf.keras.layers.experimental.preprocessing.Resizing(INCRESNET_IMG_SIZE[0],INCRESNET_IMG_SIZE[1])(input)

# EFFICIENT NET
effnet7_input = effnet7_preprocess(input_effnet7)
x_effnet7 = effnet7_model(effnet7_input)
x_effnet7 = tf.keras.layers.GlobalAveragePooling2D(name = 'global_average_layer_effnet7')(x_effnet7)

# NASNET LARGE
nasnet_input = nasnet_preprocess(input_nasnet)
x_nasnet = nasnet_model(nasnet_input)
x_nasnet = tf.keras.layers.GlobalAveragePooling2D(name = 'global_average_layer_nasnet')(x_nasnet)

# INCEPTION RES NET V2
incresnet_input = incresnet_preprocess(input_incresnet)
x_incresnet = incresnet_model(incresnet_input)
x_incresnet = tf.keras.layers.GlobalAveragePooling2D(name = 'global_average_layer_incresnet')(x_incresnet)

x = tf.keras.layers.concatenate([x_effnet7, x_nasnet, x_incresnet])

output = tf.keras.layers.Dense(NUM_CLASSES, activation='softmax')(x)

concatenate_model = tf.keras.Model(inputs=input, outputs = output)
```

    Downloading data from https://storage.googleapis.com/keras-applications/efficientnetb7_notop.h5
    258080768/258076736 [==============================] - 2s 0us/step
    Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/nasnet/NASNet-large-no-top.h5
    343613440/343610240 [==============================] - 4s 0us/step
    Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/inception_resnet_v2/inception_resnet_v2_weights_tf_dim_ordering_tf_kernels_notop.h5
    219062272/219055592 [==============================] - 2s 0us/step

```python
concatenate_model.summary()
```

    Model: "model"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_4 (InputLayer)            [(None, 256, 256, 3) 0
    __________________________________________________________________________________________________
    resizing_1 (Resizing)           (None, 331, 331, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    resizing_2 (Resizing)           (None, 299, 299, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    tf.math.truediv (TFOpLambda)    (None, 331, 331, 3)  0           resizing_1[0][0]
    __________________________________________________________________________________________________
    tf.math.truediv_1 (TFOpLambda)  (None, 299, 299, 3)  0           resizing_2[0][0]
    __________________________________________________________________________________________________
    resizing (Resizing)             (None, 600, 600, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    tf.math.subtract (TFOpLambda)   (None, 331, 331, 3)  0           tf.math.truediv[0][0]
    __________________________________________________________________________________________________
    tf.math.subtract_1 (TFOpLambda) (None, 299, 299, 3)  0           tf.math.truediv_1[0][0]
    __________________________________________________________________________________________________
    efficientnetb7 (Functional)     (None, None, None, 2 64097687    resizing[0][0]
    __________________________________________________________________________________________________
    NASNet (Functional)             (None, 11, 11, 4032) 84916818    tf.math.subtract[0][0]
    __________________________________________________________________________________________________
    inception_resnet_v2 (Functional (None, None, None, 1 54336736    tf.math.subtract_1[0][0]
    __________________________________________________________________________________________________
    global_average_layer_effnet7 (G (None, 2560)         0           efficientnetb7[0][0]
    __________________________________________________________________________________________________
    global_average_layer_nasnet (Gl (None, 4032)         0           NASNet[0][0]
    __________________________________________________________________________________________________
    global_average_layer_incresnet  (None, 1536)         0           inception_resnet_v2[0][0]
    __________________________________________________________________________________________________
    concatenate_4 (Concatenate)     (None, 8128)         0           global_average_layer_effnet7[0][0
                                                                     global_average_layer_nasnet[0][0]
                                                                     global_average_layer_incresnet[0]
    __________________________________________________________________________________________________
    dense (Dense)                   (None, 125)          1016125     concatenate_4[0][0]
    ==================================================================================================
    Total params: 204,367,366
    Trainable params: 1,016,125
    Non-trainable params: 203,351,241
    __________________________________________________________________________________________________

```python
from tensorflow.keras.utils import plot_model
plot_model(concatenate_model)
```

Es normal que el input de efficient net no haga nada, ya que este modelo pre entrenado no necesita mucha operacion, esto puede desprenderse de la documentación:
> This method does nothing and only kept as a placeholder to align the API surface between old and new version of model.

[preprocess input effnet documentation](https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet/preprocess_input)

```python
concatenate_model.compile(loss='categorical_crossentropy',
              optimizer=tf.keras.optimizers.Adam(),
              metrics=["accuracy"])
```

```python
concatenate_model.fit(train_data,
                      epochs=5,
                      steps_per_epoch=len(train_data),
                      validation_data=test_data,
                      validation_steps=len(test_data),
                      # Add TensorBoard callback to model (callbacks parameter takes a list)
                      callbacks=[create_tensorboard_callback(dir_name="tensorflow_hub", # save experiment logs here
                                                            experiment_name="Concatenate Arquitecture")])
```

    Saving TensorBoard log files to: tensorflow_hub/Concatenate Arquitecture/20210605-004035

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

    Epoch 1/5
    269/269 [==============================] - 3385s 12s/step - loss: 0.5225 - accuracy: 0.8838 - val_loss: 0.2647 - val_accuracy: 0.9274
    Epoch 2/5
    269/269 [==============================] - 1399s 5s/step - loss: 0.1281 - accuracy: 0.9578 - val_loss: 0.2674 - val_accuracy: 0.9231
    Epoch 3/5
    269/269 [==============================] - 1399s 5s/step - loss: 0.0793 - accuracy: 0.9727 - val_loss: 0.2627 - val_accuracy: 0.9303
    Epoch 4/5
    269/269 [==============================] - 1398s 5s/step - loss: 0.0571 - accuracy: 0.9814 - val_loss: 0.2875 - val_accuracy: 0.9265
    Epoch 5/5
    269/269 [==============================] - 1399s 5s/step - loss: 0.0406 - accuracy: 0.9877 - val_loss: 0.2709 - val_accuracy: 0.9355

    <tensorflow.python.keras.callbacks.History at 0x7ffa65d54b50>

```python
concatenate_model.evaluate(test_data)
```

    66/66 [==============================] - 272s 4s/step - loss: 0.2709 - accuracy: 0.9355

    [0.27093568444252014, 0.9354838728904724]

No es mejor que efficientNet7, pero puede que generalice mejor asi que lo guardaremos.

```python
concatenate_model.save('concatenate_model.h5')
```

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
      category=CustomMaskWarning)

```python
zip_folder('tensorflow_hub')
```

Lo que haremos ahora, será primero hacer un pequeño fine tuning de este nuevo modelo, el lado negativo es que toma mucho tiempo ejecutar cada epoch, pero el tiempo puede valer la pena, hasta ahora el mejor resultado que tenemos es EfficientNet7 con un 95%. Intentemos superar esto.

**Ideas**
- Fine Tuning
- Data Augmentation
- Callbacks
- Optimizadores
- Revisar Shuffle = True

Si no mejoramos el 95%, habremos aprendido a hacer muchas cosas, y podemos intentar:

- Fine Tuning EfficentNet7
- Data Agumentation EfficientNet7
- etc...

## Fine Tune

```python
# Este modelo ya lo entrenamos
model = tf.keras.models.load_model('/content/drive/MyDrive/Colab Data/Proyecto buscomiperro/modelos/concatenate_model.h5')
```

```python
# Veamos cuantos parametros se estan entrenando por ahora
model.summary()
```

    Model: "model"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_4 (InputLayer)            [(None, 256, 256, 3) 0
    __________________________________________________________________________________________________
    resizing_1 (Resizing)           (None, 331, 331, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    resizing_2 (Resizing)           (None, 299, 299, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    tf.math.truediv (TFOpLambda)    (None, 331, 331, 3)  0           resizing_1[0][0]
    __________________________________________________________________________________________________
    tf.math.truediv_1 (TFOpLambda)  (None, 299, 299, 3)  0           resizing_2[0][0]
    __________________________________________________________________________________________________
    resizing (Resizing)             (None, 600, 600, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    tf.math.subtract (TFOpLambda)   (None, 331, 331, 3)  0           tf.math.truediv[0][0]
    __________________________________________________________________________________________________
    tf.math.subtract_1 (TFOpLambda) (None, 299, 299, 3)  0           tf.math.truediv_1[0][0]
    __________________________________________________________________________________________________
    efficientnetb7 (Functional)     (None, None, None, 2 64097687    resizing[0][0]
    __________________________________________________________________________________________________
    NASNet (Functional)             (None, 11, 11, 4032) 84916818    tf.math.subtract[0][0]
    __________________________________________________________________________________________________
    inception_resnet_v2 (Functional (None, None, None, 1 54336736    tf.math.subtract_1[0][0]
    __________________________________________________________________________________________________
    global_average_layer_effnet7 (G (None, 2560)         0           efficientnetb7[0][0]
    __________________________________________________________________________________________________
    global_average_layer_nasnet (Gl (None, 4032)         0           NASNet[0][0]
    __________________________________________________________________________________________________
    global_average_layer_incresnet  (None, 1536)         0           inception_resnet_v2[0][0]
    __________________________________________________________________________________________________
    concatenate_4 (Concatenate)     (None, 8128)         0           global_average_layer_effnet7[0][0
                                                                     global_average_layer_nasnet[0][0]
                                                                     global_average_layer_incresnet[0]
    __________________________________________________________________________________________________
    dense (Dense)                   (None, 125)          1016125     concatenate_4[0][0]
    ==================================================================================================
    Total params: 204,367,366
    Trainable params: 1,016,125
    Non-trainable params: 203,351,241
    __________________________________________________________________________________________________

```python
from keras.utils.layer_utils import count_params
p = count_params(model.trainable_weights)
print(f"Se entrenaron {p} parámetros")
```

    Se entrenaron 1016125 parámetros

Hay que descongelar los pesos de cada modelo, veamos cuantos layers tiene cada uno

```python
efficient_layer = model.layers[8]
nasnet_layer = model.layers[9]
incresnet_layer = model.layers[10]
```

```python
print(f"EfficientNet7 tiene {len(efficient_layer.layers)} layers")
print(f"NasNet Large tiene {len(nasnet_layer.layers)} layers")
print(f"Inception ResNet tiene {len(incresnet_layer.layers)} layers")
```

    EfficientNet7 tiene 813 layers
    NasNet Large tiene 1039 layers
    Inception ResNet tiene 780 layers

Hay que tener en cuenta que las capas superiores son las que extraen patrones más complejos, por lo que lo ideal seria descongelar estas. Voy a intentar descongelar el 10% superior de cada modelo

```python
# Antes veamos que no se esten entrenando parametros en estos modelos pre entrenados
print(efficient_layer.trainable_weights)
print(nasnet_layer.trainable_weights)
print(incresnet_layer.trainable_weights)
```

    []
    []
    []

```python
def unfreeze_layers(functional_layer, p):
  """"
  Función que descongela las últimas p porciento de las capas

  Args:
  functional_layer : modelo pre entrenado
  p : porcentaje a descongelar

  Return:
  None
  """

  # Descongelamos todas
  functional_layer.trainable = True
  num_trainable_layers = int(p*(len(functional_layer.layers)))

  #Freezeamos todas menos las ultimas p% capas
  for layer in functional_layer.layers[:-num_trainable_layers]:
    layer.trainable = False

  count = 0
  # Veamos ahora cuantas descongelamos
  for layer in functional_layer.layers:
    if layer.trainable == True:
      count += 1

  print(f"se han descongelado {count} layers")

```

```python
unfreeze_layers(efficient_layer, 0.1)
```

    se han descongelado 81 layers

```python
unfreeze_layers(nasnet_layer, 0.1)
```

    se han descongelado 103 layers

```python
unfreeze_layers(incresnet_layer, 0.1)
```

    se han descongelado 78 layers

```python
# Aseguremonos de que haya surtido efecto en el modelo general
count_params(model.trainable_weights)
```

    60213469

```python
model.summary()
```

    Model: "model"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_4 (InputLayer)            [(None, 256, 256, 3) 0
    __________________________________________________________________________________________________
    resizing_1 (Resizing)           (None, 331, 331, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    resizing_2 (Resizing)           (None, 299, 299, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    tf.math.truediv (TFOpLambda)    (None, 331, 331, 3)  0           resizing_1[0][0]
    __________________________________________________________________________________________________
    tf.math.truediv_1 (TFOpLambda)  (None, 299, 299, 3)  0           resizing_2[0][0]
    __________________________________________________________________________________________________
    resizing (Resizing)             (None, 600, 600, 3)  0           input_4[0][0]
    __________________________________________________________________________________________________
    tf.math.subtract (TFOpLambda)   (None, 331, 331, 3)  0           tf.math.truediv[0][0]
    __________________________________________________________________________________________________
    tf.math.subtract_1 (TFOpLambda) (None, 299, 299, 3)  0           tf.math.truediv_1[0][0]
    __________________________________________________________________________________________________
    efficientnetb7 (Functional)     (None, None, None, 2 64097687    resizing[0][0]
    __________________________________________________________________________________________________
    NASNet (Functional)             (None, 11, 11, 4032) 84916818    tf.math.subtract[0][0]
    __________________________________________________________________________________________________
    inception_resnet_v2 (Functional (None, None, None, 1 54336736    tf.math.subtract_1[0][0]
    __________________________________________________________________________________________________
    global_average_layer_effnet7 (G (None, 2560)         0           efficientnetb7[0][0]
    __________________________________________________________________________________________________
    global_average_layer_nasnet (Gl (None, 4032)         0           NASNet[0][0]
    __________________________________________________________________________________________________
    global_average_layer_incresnet  (None, 1536)         0           inception_resnet_v2[0][0]
    __________________________________________________________________________________________________
    concatenate_4 (Concatenate)     (None, 8128)         0           global_average_layer_effnet7[0][0
                                                                     global_average_layer_nasnet[0][0]
                                                                     global_average_layer_incresnet[0]
    __________________________________________________________________________________________________
    dense (Dense)                   (None, 125)          1016125     concatenate_4[0][0]
    ==================================================================================================
    Total params: 204,367,366
    Trainable params: 60,213,469
    Non-trainable params: 144,153,897
    __________________________________________________________________________________________________

FIUUUU, tendremos que ajustar aproximadamente 59 millones más de parámetros!!!!

Vamos a intentar fitear este modelo, si se demora mucho, tendremos que optar por simplemente tunear EfficientNet7.
Recordar que Keras salva modelos inspeccionando sus arquitecturas. Esta técnica lo salva todo:

- Los valores de peso
- La arquitectura del modelo
- La configuración de entrenamiento del modelo (lo que pasa al método .compile() )
- El optimizador y su estado, si lo hay (esto le permite reiniciar el entrenamiento donde lo dejó)

Por lo que solo tenemos que hacer el `model.fit()`

```python
model.fit(train_data,
          epochs=5,
          steps_per_epoch=len(train_data),
          validation_data=test_data,
          validation_steps=len(test_data))
```

# Conclusión :
En cuanto a performance y tiempo, conviene más hacer fine tuning al modelo pre entrenado EfficientNet. Eso lo realizaré en el notebook `BPM_ModelV2.py`

```python

```


```python
import sys
sys.path.append('..')
import argparse
import os
import torch
import numpy as np
import matplotlib.pyplot as plt
from sympy import simplify_logic
import time
from sklearn.metrics import accuracy_score
import pandas as pd
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.tree import _tree, export_text
from torchvision import datasets, transforms
from torch.utils.data import TensorDataset, DataLoader
from torch.utils.data.sampler import SubsetRandomSampler
import torch.nn.functional as F
from sklearn.preprocessing import OneHotEncoder

from lens.utils.base import validate_network, set_seed, tree_to_formula
from lens.utils.relunn import get_reduced_model, prune_features
from lens.utils.sigmoidnn import prune_equal_fanin
from lens import logic

from dSprites.dSprites_loader import load_dsprites, concept_filters, get_shape_scale
from dSprites.dSprites_style_I2C import i2c_style

results_dir = './results/dsprites'
if not os.path.isdir(results_dir):
    os.makedirs(results_dir)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
n_rep = 10
tot_epochs = 10001
prune_epochs = 60001
```

```python
def get_data(path):
    # Load dataset specified by schema
    dataset_schema = 'small_skip'
    # dataset_schema = 'big_skip'

    # Get filtered concept values
    shape_range, scale_range, rot_range, x_pos_range, y_pos_range = concept_filters(dataset_schema)

    # Define function for filtering out specified concept values only
    def c_filter_fn(concepts):
        in_shape_range = (concepts[1] in shape_range)
        in_scale_range = (concepts[2] in scale_range)
        in_rot_range = (concepts[3] in rot_range)
        in_x_range = (concepts[4] in x_pos_range)
        in_y_range = (concepts[5] in y_pos_range)

        return (in_shape_range and in_scale_range and in_rot_range and in_x_range and in_y_range)

    label_fn = get_shape_scale(shape_range, scale_range)

    # Load dataset
    x_train, y_train, x_val, y_val, x_test, y_test, c_train, c_val, c_test, c_names = load_dsprites(path,
                                                                               c_filter_fn=c_filter_fn,
                                                                               label_fn=label_fn,
                                                                               train_test_split_flag=True)

    return x_train, y_train, x_val, y_val, x_test, y_test, c_train, c_val, c_test, c_names
```

```python
# Load the model, as well as input, label, and concept data
data = get_data(path='./dSprites/data/dsprites.npz')
x_train, y_train, x_val, y_val, x_test, y_test, c_train, c_val, c_test, c_names = data
x_train = torch.FloatTensor(x_train)
y_train = torch.FloatTensor(y_train)
x_val = torch.FloatTensor(x_val)
y_val = torch.FloatTensor(y_val)
x_test = torch.FloatTensor(x_test)
y_test = torch.FloatTensor(y_test)
c_train = torch.FloatTensor(c_train)
c_val = torch.FloatTensor(c_val)
c_test = torch.FloatTensor(c_test)
print("Data loaded successfully...")
```

    x_train shape: (20993, 3, 64, 64)
    c_train shape: (20993, 50)
    y_train shape: (20993, 18)
    Number of images in x_train 20993
    Number of images in x_val 10341
    Number of images in x_test 5530
    Data loaded successfully...

```python
args = {
    'models_dir': './models/dSprites/',
    'model_style': 'CBM',
    'seed': 0,
    'batch_size': 128,
}

if not os.path.isdir(args['models_dir']):
    os.makedirs(args['models_dir'])
```

```python
train_dataset = TensorDataset(x_train, c_train)
train_dataloader = DataLoader(train_dataset, batch_size=args['batch_size'], shuffle=False)
val_dataset = TensorDataset(x_val, c_val)
val_dataloader = DataLoader(val_dataset, batch_size=args['batch_size'], shuffle=False)
test_dataset = TensorDataset(x_test, c_test)
test_dataloader = DataLoader(test_dataset, batch_size=args['batch_size'], shuffle=False)

c_predictions_train, c_predictions_val, c_predictions_test = i2c_style(train_dataloader, val_dataloader, test_dataloader, device, args)
c_predictions_train = torch.FloatTensor(c_predictions_train)
c_predictions_val = torch.FloatTensor(c_predictions_val)
c_predictions_test = torch.FloatTensor(c_predictions_test)
```

```python
x_train = c_val
x_test = c_test
x_train.shape
```

    torch.Size([10341, 50])

```python
y_train = y_val[:, 0].to(torch.long)
y_test = y_test[:, 0].to(torch.long)
y_train.shape
```

    torch.Size([10341])

```python
colors = ['white']
shapes = ['square', 'ellipse', 'heart']
scale = ['very small', 'small', 's-medium', 'b-medium', 'big', 'very big']
rotation = ['0°', '5°', '10°', '15°', '20°', '25°', '30°', '35°']
x_pos = ['x0', 'x2', 'x4', 'x6', 'x8', 'x10', 'x12', 'x14', 'x16', 'x18', 'x20', 'x22', 'x24', 'x26', 'x28', 'x30']
y_pos = ['y0', 'y2', 'y4', 'y6', 'y8', 'y10', 'y12', 'y14', 'y16', 'y18', 'y20', 'y22', 'y24', 'y26', 'y28', 'y30']
concepts = colors + shapes + scale + rotation + x_pos + y_pos
len(concepts)
```

    50

```python
def train_nn(x_train, y_train, need_pruning, seed, device, relu=False, verbose=False):
    set_seed(seed)
    x_train = x_train.to(device)
    y_train = y_train.to(device)
    layers = [
        torch.nn.Linear(x_train.size(1), 30),
#         torch.nn.ReLU() if relu else torch.nn.LeakyReLU(),
#         torch.nn.Linear(100, 50),
        torch.nn.ReLU() if relu else torch.nn.LeakyReLU(),
        torch.nn.Linear(30, 10),
        torch.nn.ReLU() if relu else torch.nn.LeakyReLU(),
        torch.nn.Linear(10, 2),
        torch.nn.Softmax(dim=1),
    ]
    model = torch.nn.Sequential(*layers).to(device)

    optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001)
    loss_form = torch.nn.CrossEntropyLoss()
    model.train()
    for epoch in range(tot_epochs):
        # forward pass
        optimizer.zero_grad()
        y_pred = model(x_train)
        # Compute Loss
        loss = loss_form(y_pred, y_train)

        for module in model.children():
            if isinstance(module, torch.nn.Linear):
                loss += 0.0007 * torch.norm(module.weight, 1)
                loss += 0.0007 * torch.norm(module.bias, 1)
                break

        # backward pass
        loss.backward()
        optimizer.step()

        if epoch > prune_epochs and need_pruning:
            prune_features(model, n_classes=1, device=device)
            need_pruning = False

        # compute accuracy
        if epoch % 500 == 0 and verbose:
            y_pred_d = torch.argmax(y_pred, dim=1)
            accuracy = y_pred_d.eq(y_train).sum().item() / y_train.size(0)
            print(f'\t Epoch {epoch}: train accuracy: {accuracy:.4f}')

    return model
```

```python
def train_psi_nn(x_train, y_train, need_pruning, seed, device, verbose=False):
    set_seed(seed)
    x_train = x_train.to(device)
    y_train = y_train.to(device).to(torch.float)
    layers = [
        torch.nn.Linear(x_train.size(1), 10),
        torch.nn.Sigmoid(),
        torch.nn.Linear(10, 4),
        torch.nn.Sigmoid(),
        torch.nn.Linear(4, 1),
        torch.nn.Sigmoid(),
    ]
    model = torch.nn.Sequential(*layers).to(device)

    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    loss_form = torch.nn.BCELoss()
    model.train()
    for epoch in range(tot_epochs):
        # forward pass
        optimizer.zero_grad()
        y_pred = model(x_train).squeeze()
        # Compute Loss
        loss = loss_form(y_pred, y_train)

        for module in model.children():
            if isinstance(module, torch.nn.Linear):
                loss += 0.00001 * torch.norm(module.weight, 1)

        # backward pass
        loss.backward()
        optimizer.step()

        if epoch > prune_epochs and need_pruning:
            model = prune_equal_fanin(model, 2, validate=True, device=device)
            need_pruning = False

        # compute accuracy
        if epoch % 500 == 0 and verbose:
            y_pred_d = y_pred > 0.5
            accuracy = y_pred_d.eq(y_train).sum().item() / y_train.size(0)
            print(f'\t Epoch {epoch}: train accuracy: {accuracy:.4f}')

    return model
```

```python
def c_to_y(method, need_pruning, relu, verbose=False):
    methods = []
    splits = []
    explanations = []
    explanations_inv = []
    model_accuracies = []
    explanation_accuracies = []
    explanation_accuracies_inv = []
    elapsed_times = []
    elapsed_times_inv = []
    for seed in range(n_rep):
        explanation, explanation_inv = '', ''
        explanation_accuracy, explanation_accuracy_inv = 0, 0
        elapsed_time, elapsed_time_inv = 0, 0

        print(f'Seed [{seed+1}/{n_rep}]')

        if method == 'tree':
            classifier = DecisionTreeClassifier(random_state=seed)
            classifier.fit(x_train.detach().numpy(), y_train.detach().numpy())
            y_preds = classifier.predict(x_test.detach().numpy())
            model_accuracy = accuracy_score(y_test.detach().numpy(), y_preds)

            target_class = 1
            start = time.time()
            explanation = tree_to_formula(classifier, concepts, target_class)
            elapsed_time = time.time() - start

            target_class_inv = 0
            start = time.time()
            explanation_inv = tree_to_formula(classifier, concepts, target_class_inv)
            elapsed_time = time.time() - start

        else:
            if method == 'psi':
                # positive class
                target_class = 1
                model = train_psi_nn(x_train, y_train.eq(target_class), need_pruning, seed, device, verbose)
                y_preds = model(x_test.to(device)).cpu().detach().numpy()
                model_accuracy = accuracy_score(y_test.cpu().detach().numpy(), y_preds>0.5)

            else:
                model = train_nn(x_train, y_train, need_pruning, seed, device, relu, verbose)
                y_preds = model(x_test.to(device)).cpu().detach().numpy()
                model_accuracy = accuracy_score(y_test.cpu().detach().numpy(), y_preds.argmax(axis=1))

            # positive class
            target_class = 1
            start = time.time()
            if method == 'psi':
                global_explanation = logic.generate_fol_explanations(model, device)[0]
            else:
                global_explanation, _, _ = logic.relunn.combine_local_explanations(model,
                                                                                x_train.to(device), y_train.to(device),
                                                                                target_class=target_class,
                                                                                method=method, device=device)
            elapsed_time = time.time() - start

            if global_explanation:
                explanation_accuracy, _ = logic.base.test_explanation(global_explanation, target_class, x_test, y_test)
                explanation = logic.base.replace_names(global_explanation, concepts)

#             # negative class
#             target_class_inv = 0
#             if method == 'psi':
#                 model = train_psi_nn(x_train, y_train.eq(target_class_inv), need_pruning, seed, device, verbose)

#             start = time.time()
#             if method == 'psi':
#                 global_explanation_inv = logic.generate_fol_explanations(model, device)[0]
#             else:
#                 global_explanation_inv, _, _ = logic.relunn.combine_local_explanations(model,
#                                                                                x_train.to(device),
#                                                                                y_train.to(device),
#                                                                                target_class=target_class_inv,
#                                                                                method=method, device=device)
#             elapsed_time_inv = time.time() - start
#             if global_explanation_inv:
#                 explanation_accuracy_inv, _ = logic.base.test_explanation(global_explanation_inv,
#                                                                           target_class_inv, x_test, y_test)
#                 explanation_inv = logic.base.replace_names(global_explanation_inv, concepts)

        if verbose:
            print(f'\t Model\'s accuracy: {model_accuracy:.4f}')
            print(f'\t Class {target_class} - Global explanation: "{explanation}" - Accuracy: {explanation_accuracy:.4f}')
            print(f'\t Elapsed time {elapsed_time}')
#             print(f'\t Class {target_class} - Global explanation: "{explanation_inv}" - Accuracy: {explanation_accuracy_inv:.4f}')
#             print(f'\t Elapsed time {elapsed_time_inv}')

        methods.append(method)
        splits.append(seed)
        explanations.append(explanation)
        explanations_inv.append(explanation_inv)
        model_accuracies.append(model_accuracy)
        explanation_accuracies.append(explanation_accuracy)
        explanation_accuracies_inv.append(explanation_accuracy_inv)
        elapsed_times.append(elapsed_time)
        elapsed_times_inv.append(elapsed_time_inv)

    results = pd.DataFrame({
        'method': methods,
        'split': splits,
        'explanation': explanations,
        'explanation_inv': explanations_inv,
        'model_accuracy': model_accuracies,
        'explanation_accuracy': explanation_accuracies,
        'explanation_accuracy_inv': explanation_accuracies_inv,
        'elapsed_time': elapsed_times,
        'elapsed_time_inv': elapsed_times_inv,
    })
    results.to_csv(os.path.join(results_dir, f'results_{method}.csv'))

    return results
```

# General pruning

```python
method = 'pruning'
need_pruning = True
relu = False
results_pruning = c_to_y(method, need_pruning, relu, True)
results_pruning
```

# LIME

```python
# method = 'lime'
# need_pruning = False
# relu = False
# results_lime = c_to_y(method, need_pruning, relu)
# results_lime
```

# ReLU

```python
method = 'weights'
need_pruning = False
relu = True
results_weights = c_to_y(method, need_pruning, relu)
results_weights
```

# Psi network

```python
method = 'psi'
need_pruning = True
relu = False
results_psi = c_to_y(method, need_pruning, relu, verbose=True)
results_psi
```

# Decision tree

```python
method = 'tree'
need_pruning = False
relu = False
results_tree = c_to_y(method, need_pruning, relu, True)
results_tree
```

# Summary

```python
cols = ['model_accuracy', 'explanation_accuracy', 'explanation_accuracy_inv', 'elapsed_time', 'elapsed_time_inv']
mean_cols = [f'{c}_mean' for c in cols]
sem_cols = [f'{c}_sem' for c in cols]

# pruning
df_mean = results_pruning[cols].mean()
df_sem = results_pruning[cols].sem()
df_mean.columns = mean_cols
df_sem.columns = sem_cols
summary_pruning = pd.concat([df_mean, df_sem])
summary_pruning.name = 'pruning'

# # lime
# df_mean = results_lime[cols].mean()
# df_sem = results_lime[cols].sem()
# df_mean.columns = mean_cols
# df_sem.columns = sem_cols
# summary_lime = pd.concat([df_mean, df_sem])
# summary_lime.name = 'lime'

# weights
df_mean = results_weights[cols].mean()
df_sem = results_weights[cols].sem()
df_mean.columns = mean_cols
df_sem.columns = sem_cols
summary_weights = pd.concat([df_mean, df_sem])
summary_weights.name = 'weights'

# psi
df_mean = results_psi[cols].mean()
df_sem = results_psi[cols].sem()
df_mean.columns = mean_cols
df_sem.columns = sem_cols
summary_psi = pd.concat([df_mean, df_sem])
summary_psi.name = 'psi'

# tree
df_mean = results_tree[cols].mean()
df_sem = results_tree[cols].sem()
df_mean.columns = mean_cols
df_sem.columns = sem_cols
summary_tree = pd.concat([df_mean, df_sem])
summary_tree.name = 'tree'

summary = pd.concat([summary_pruning,
#                      summary_lime,
                     summary_weights, summary_psi, summary_tree], axis=1).T
summary.columns = mean_cols + sem_cols
summary
```

```python
summary.to_csv(os.path.join(results_dir, 'summary.csv'))
```