```python
%matplotlib inline
```

Video processing, Object detection & Tracking
==============================================

**Demonstrating the video processing capabilities of Stone Soup**

This notebook will guide you progressively through the steps necessary to:

1. Use the Stone Soup :class:`~.FrameReader` components to open and process video data;
2. Use the :class:`~.TensorFlowBoxObjectDetector` to detect objects in video data, making use of Tensorflow object detection models;
3. Build a :class:`~.MultiTargetTracker` to perform tracking of multiple object in video data.

Software dependencies
---------------------
Before we begin with this tutorial, there are a few things that we need to install in order to
proceed.

FFmpeg
~~~~~~
FFmpeg is a free and open-source project consisting of a vast software suite of libraries and
programs for handling video, audio, and other multimedia files and streams. Stone Soup (or more
accurately some of its extra dependencies) make use of FFmpeg to read and output video. Download
links and installation instructions for FFmpeg can be found `here <https://www.ffmpeg.org/download.html>`__.

TensorFlow
~~~~~~~~~~
TensorFlow is a free and open-source software library for dataflow and differentiable programming
across a range of tasks, such machine learning. TensorFlow includes an Object Detection API that
makes it easy to construct, train and deploy object detection models, as well as a collection of
pre-trained models that can be used for out-of-the-box inference. A quick TensorFlow installation
tutorial can be found `here <https://tensorflow2objectdetectioninstallation.readthedocs.io/en/latest/>`__.

Stone Soup
~~~~~~~~~~
To perform video-processing using Stone Soup, we need to install some extra dependencies. The
easiest way to achieve this is by running the following commands in a Terminal window:

.. code::

    git clone "https://github.com/dstl/Stone-Soup.git"
    cd Stone-Soup
    python -m pip install -e .[dev,video,tensorflow]

Pytube
~~~~~~
We will also use pytube_ to download a Youtube video for the purposes of this tutorial. In the
same Terminal window, run the following command to install ``pytube``:

.. code::

    pip install pytube3

Using the Stone Soup :class:`~.FrameReader` classes
---------------------------------------------------
The :class:`~.FrameReader` abstract class is intended as the base class for Stone Soup readers
that read frames from any form of imagery data. As of now, Stone Soup has two implementations of
:class:`~.FrameReader` subclasses:

1. The :class:`~.VideoClipReader` component, which uses MoviePy_ to read video frames from a file.
2. The :class:`~.FFmpegVideoStreamReader` component, which uses ffmpeg-python_ to read frames from real-time video streams (e.g. RTSP).

In this tutorial we will focus on the :class:`~.VideoClipReader`, since setting up a stream for
the :class:`~.FFmpegVideoStreamReader` is more involved. Nevertheless, the use and interface of
the two readers is mostly identical after initialisation and an example of how to initialise the
later will also be provided

Download and store the video
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
First we will download the video that we will use throughout this tutorial. The code snippet
shown bellow will download the video and save it your working directory as ``sample1.mp4``.

```python
import os
from pytube import YouTube
VIDEO_FILENAME = 'sample1'
VIDEO_EXTENTION = '.mp4'
VIDEO_PATH = os.path.join(os.getcwd(), VIDEO_FILENAME+VIDEO_EXTENTION)

if not os.path.exists(VIDEO_PATH):
    yt = YouTube('http://www.youtube.com/watch?v=MNn9qKG2UFI')
    yt.streams[0].download(filename=VIDEO_FILENAME)
```

Building the video reader
~~~~~~~~~~~~~~~~~~~~~~~~~

VideoClipReader
***************
We will use the :class:`~.VideoClipReader` class to read and replay the downloaded file. We also
configure the reader to only replay the clip for the a duration of 2 seconds between `00:10` and
`00:12`.

```python
import datetime
from stonesoup.reader.video import VideoClipReader
start_time = datetime.timedelta(minutes=0, seconds=10)
end_time = datetime.timedelta(minutes=0, seconds=12)
frame_reader = VideoClipReader(VIDEO_PATH, start_time, end_time)
```

It is also possible to apply clip transformations and effects, as per the
`MoviePy documentation <https://zulko.github.io/moviepy/getting_started/effects.html>`_.
The underlying MoviePy :class:`~VideoFileClip` instance can be accessed through the
:attr:`~.VideoClipReader.clip` class property. For example, we can crop out 100 pixels from
the top and left of the frames, as they are read by the reader, as shown below.

```python
from moviepy.video.fx import all
frame_reader.clip = all.crop(frame_reader.clip, 100, 100)
num_frames = len(list(frame_reader.clip.iter_frames()))
```

FFmpegVideoStreamReader
***********************
For reference purposes, we also include here an example of how to build a
:class:`~.FFmpegVideoStreamReader`. Let's assume that we have a camera which broadcasts its feed
through a public RTSP stream, under the URL ``rtsp://192.168.55.10:554/stream``. We can build a
:class:`~.FFmpegVideoStreamReader` object to read frames from this stream as follows:

.. code:: python

  in_opts = {'threads': 1, 'fflags': 'nobuffer'}
  out_opts = {'format': 'rawvideo', 'pix_fmt': 'bgr24'}
  stream_url = 'rtsp://192.168.55.10:554/stream'
  video_reader = FFmpegVideoStreamReader(stream_url, input_opts=in_opts, output_opts=out_opts)

.. important::

  Note that the above code is an illustrative example and will not be run.

:attr:`~.FFmpegVideoStreamReader.input_opts` and :attr:`~.FFmpegVideoStreamReader.output_opts`
are optional arguments, which allow users to specify options for the input and output FFmpeg
streams, as documented by `FFmpeg <https://ffmpeg.org/ffmpeg.html#toc-Options>`__ and
ffmpeg-python_.

Reading frames from the reader
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All :class:`~.FrameReader` objects, of which the :class:`~.VideoClipReader` is a subclass,
generate frames in the form of :class:`~.ImageFrame` objects. Below we show an example of how to
read and visualise these frames using `matplotlib`.

```python
from copy import copy
from PIL import Image
from matplotlib import pyplot as plt
from matplotlib import animation

fig, ax = plt.subplots(num="VideoClipReader output")
artists = []

print('Running FrameReader example...')
for timestamp, frame in frame_reader:
    if not (len(artists)+1) % 10:
        print("Frame: {}/{}".format(len(artists)+1, num_frames))

    # Read the frame pixels
    pixels = copy(frame.pixels)

    # Plot output
    image = Image.fromarray(pixels)
    ax.axes.xaxis.set_visible(False)
    ax.axes.yaxis.set_visible(False)
    fig.tight_layout()
    artist = ax.imshow(image, animated=True)
    artists.append([artist])

ani = animation.ArtistAnimation(fig, artists, interval=20, blit=True, repeat_delay=200)
```

Using the :class:`~.TensorFlowBoxObjectDetector` class
------------------------------------------------------
We now continue by demonstrating how to use the :class:`~.TensorFlowBoxObjectDetector` to detect
objects, and more specifically cars, within the frames read in by our ``video_reader``. The
:class:`~.TensorFlowBoxObjectDetector` can utilise both pre-trained and custom-trained TensorFlow
object detection models which generate detection in the form of bounding boxes. In this example,
we will make use of a pre-trained model from the
`TensorFlow detection model zoo <https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md>`_,
but the process of using a custom-trained TensorFlow model is the same.

Downloading the model
~~~~~~~~~~~~~~~~~~~~~
The code snippet shown below is used to download the object detection model checkpoint file
(.pb) that we will feed into the :class:`~.TensorFlowBoxObjectDetector` , as well as the label
file (.pbtxt) which contains a list of strings used to add the correct label to each detection
(e.g. car).

The particular detection algorithm we will use is the Faster-RCNN, with an Inception
Resnet v2 backbone and running in Atrous mode with low proposals, pre-trained on the MSCOCO
dataset.

Warning

**The downloaded model has a size of approximately 500 MB**. Therefore it is advised that you
  run the script on a stable (ideally not mobile) internet connection. The files will only be
  downloaded the first time the script is run. In consecutive runs the code will skip this step,
  provided that ``PATH_TO_CKPT`` and ``PATH_TO_LABELS`` are valid paths.

```python
import urllib
import tarfile
MODEL_NAME = 'faster_rcnn_inception_resnet_v2_atrous_lowproposals_coco_2018_01_28'
MODEL_TAR = MODEL_NAME + '.tar.gz'
MODELS_DOWNLOAD_BASE = 'http://download.tensorflow.org/models/object_detection/'
PATH_TO_CKPT = os.path.join(os.getcwd(), MODEL_NAME + '/frozen_inference_graph.pb')

# Download and extract model
if not os.path.exists(PATH_TO_CKPT):
    print("Downloading model. This may take a while...")
    urllib.request.urlretrieve(MODELS_DOWNLOAD_BASE + MODEL_TAR, MODEL_TAR)
    tar_file = tarfile.open(MODEL_TAR)
    for file in tar_file.getmembers():
        file_name = os.path.basename(file.name)
        if 'frozen_inference_graph.pb' in file_name:
            tar_file.extract(file, os.getcwd())
    tar_file.close()
    os.remove(MODEL_TAR)

LABEL_FILE = 'mscoco_label_map.pbtxt'
LABELS_DOWNLOAD_BASE = \
    'https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/'
PATH_TO_LABELS = os.path.join(os.getcwd(), "{}/{}".format(MODEL_NAME, LABEL_FILE))

# Download labels
if not os.path.exists(PATH_TO_LABELS):
    print("Downloading label file...")
    urllib.request.urlretrieve(LABELS_DOWNLOAD_BASE + LABEL_FILE, PATH_TO_LABELS)
```

Building the detector
~~~~~~~~~~~~~~~~~~~~~
Next, we proceed to initialise our detector object. To do this, we require the ``frame_reader``
object we built previously, as well as a path to the (downloaded) model checkpoint (.pb) and
label (.pbtxt) files, which we have already defined above under the ``PATH_TO_CKPT`` and
``PATH_TO_LABELS`` variables.

The :class:`~.TensorFlowBoxObjectDetector` object can optionally be configured to digest frames
from the provided reader asynchronously, and only perform detection on the last frame digested,
by setting ``run_async=True``.This is suitable when the detector is applied to readers generating
a live feed (e.g. the :class:`~.FFmpegVideoStreamReader`), where real-time processing is
paramount. Since we are using a :class:`~.VideoClipReader` in this example, we set
``run_async=False``, which is also the default setting.

Finally, the ``session_config`` parameter can be used to provide a `ConfigProto
<https://www.tensorflow.org/code/tensorflow/core/protobuf/config.proto>`_ protocol buffer with
configuration options for the TensorFlow session run by the detector. Below we show an example of
how this can be used to prevent TensorFlow from mapping all of the GPU memory, which can lead to
cuDNN errors when the host GPU is used by other processes.

```python
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'    # Suppress TensorFlow logging
import tensorflow as tf
from stonesoup.detector.tensorflow import TensorFlowBoxObjectDetector

config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True      # Allow dynamic allocation of GPU memory to TF session
run_async = False                           # Configure the detector to run in synchronous mode
detector = TensorFlowBoxObjectDetector(frame_reader, PATH_TO_CKPT, PATH_TO_LABELS,
                                       run_async=run_async, session_config=config)
```

Filtering-out unwanted detections
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this section we showcase how we can utilise Stone Soup :class:`~.Feeder` objects in order to
filter out unwanted detections. One example of feeder we can use is the
:class:`~.MetadataValueFilter`, which allows us to filter detections by applying a custom
operator on particular fields of the :attr:`~.Detection.metadata` property of detections.

Each detection generated by :class:`~.TensorFlowBoxObjectDetector` carries the following
:attr:`~.Detection.metadata` fields:

 - ``raw_box``: The raw bounding box containing the normalised coordinates ``[y_0, x_0, y_1, x_1]``, as generated by TensorFlow.
 - ``class``: A dict with keys ``id`` and ``name`` relating to the id and name of the detection class.
 - ``score``: A float in the range ``(0, 1]`` indicating the detector's confidence.

Detection models trained on the MSCOCO dataset, such as the one we downloaded, are able to detect
90 different classes of objects (see the `downloaded .pbtxt file <https://github.com/tensorflow/models/blob/master/research/object_detection/data/mscoco_label_map.pbtxt>`_
for a full list). Instead, as we discussed at the beginning of the tutorial, we wish to limit the
detections to only those classified as cars. This can be done as follows:

```python
from stonesoup.feeder.filter import MetadataValueFilter
detector = MetadataValueFilter(detector, 'class', lambda x: x['name'] == 'car')
```

Continuing, we may want to filter out detections which have a low confidence score:

```python
detector = MetadataValueFilter(detector, 'score', lambda x: x > 0.1)
```

Finally, we observed that the detector tends to incorrectly generate detections which are much
larger the the size we expect for a car. Therefore, we can filter out those detections by only
allowing ones whose width is less the 20\% of the frame width (i.e. ``x_1-x_0 < 0.2``):

```python
detector = MetadataValueFilter(detector, 'raw_box', lambda x: x[3]-x[1] < 0.2)
```

You are encouraged to comment out any/all of the above filter definitions and observe the
produced output.

Reading and visualising detections
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Detections generated by the :class:`~.TensorFlowBoxObjectDetector` have a 4-dimensional
:attr:`~.Detection.state_vector` in the form of a bounding boxes that captures the area of the
frame where an object is detected. Each bounding box is represented by a vector of the form
``[x, y, w, h]``, where ``x, y`` denote the relative pixel coordinates of the top-left corner,
while ``w, h`` denote the relative width and height of the bounding box. Below we show an example
of how to read and visualise these detections using `matplotlib`.

```python
import numpy as np
from PIL import ImageDraw

def draw_detections(image, detections, show_class=False, show_score=False):
    """ Draw detections on an image

    Parameters
    ----------
    image: :class:`PIL.Image`
        Image on which to draw the detections
    detections: : set of :class:`~.Detection`
        A set of detections generated by :class:`~.TensorFlowBoxObjectDetector`
    show_class: bool
        Whether to draw the class of the object. Default is ``False``
    show_score: bool
        Whether to draw the score of the object. Default is ``False``

    Returns
    -------
    : :class:`PIL.Image`
        Image with detections drawn
    """
    draw = ImageDraw.Draw(image)
    for detection in detections:
        x0, y0, w, h = np.array(detection.state_vector).reshape(4)
        x1, y1 = (x0 + w, y0 + h)
        draw.rectangle([x0, y0, x1, y1], outline=(0, 255, 0), width=1)
        class_ = detection.metadata['class']['name']
        score = round(float(detection.metadata['score']),2)
        if show_class and show_score:
            draw.text((x0,y1 + 2), '{}:{}'.format(class_, score), fill=(0, 255, 0))
        elif show_class:
            draw.text((x0, y1 + 2), '{}'.format(class_), fill=(0, 255, 0))
        elif show_score:
            draw.text((x0, y1 + 2), '{}'.format(score), fill=(0, 255, 0))

    del draw
    return image

fig2, ax2 = plt.subplots(num="TensorFlowBoxObjectDetector output")
artists2 = []
print("Running TensorFlowBoxObjectDetector example... Be patient...")
for timestamp, detections in detector:
    if not (len(artists2)+1) % 10:
        print("Frame: {}/{}".format(len(artists2)+1, num_frames))

    # Read the frame pixels
    frame = frame_reader.frame
    pixels = copy(frame.pixels)

    # Plot output
    image = Image.fromarray(pixels)
    image = draw_detections(image, detections, True, True)
    ax2.axes.xaxis.set_visible(False)
    ax2.axes.yaxis.set_visible(False)
    fig2.tight_layout()
    artist = ax2.imshow(image, animated=True)
    artists2.append([artist])

ani2 = animation.ArtistAnimation(fig2, artists2, interval=20, blit=True, repeat_delay=200)
```

Constructing a Multi-Object Video Tracker
-----------------------------------------
In this final segment of the tutorial we will see how we can use the above demonstrated
components to perform tracking of multiple objects within Stone Soup.

Defining the state-space models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Transition Model
****************
We begin our definition of the state-space models by defining the hidden state
$\mathrm{x}_k$, i.e. the state that we wish to estimate:

\begin{align}\mathrm{x}_k = [x_k, \dot{x}_k, y_k, \dot{y}_k, w_k, h_k]\end{align}

where $x_k, y_k$ denote the pixel coordinates of the top-left corner of the bounding box
containing an object, with $\dot{x}_k, \dot{y}_k$ denoting their respective rate of change,
while $w_k$ and $h_k$ denote the width and height of the box, respectively.

We assume that $x_k$ and $y_k$ move with nearly :class:`~.ConstantVelocity`, while
$w_k$ and $h_k$ evolve according to a :class:`~.RandomWalk`.Using these assumptions,
we proceed to construct our Stone Soup :class:`~.TransitionModel` as follows:

```python
from stonesoup.models.transition.linear import (CombinedLinearGaussianTransitionModel,
                                                ConstantVelocity, RandomWalk)
t_models = [ConstantVelocity(20**2), ConstantVelocity(20**2), RandomWalk(20**2), RandomWalk(20**2)]
transition_model = CombinedLinearGaussianTransitionModel(t_models)
```

Measurement Model
*****************
Continuing, we define the measurement state $\mathrm{y}_k$, which follows naturally from
the form of the detections generated by the :class:`~.TensorFlowBoxObjectDetector` we previously
discussed:

\begin{align}\mathrm{y}_k = [x_k, y_k, w_k, h_k]\end{align}

We make use of a 4-dimensional :class:`~.LinearGaussian` model as our :class:`~.MeasurementModel`,
whereby we can see that the individual indices of $\mathrm{y}_k$ map to indices `[0,2,4,5]`
of the 6-dimensional state $\mathrm{x}_k$:

```python
from stonesoup.models.measurement.linear import LinearGaussian
measurement_model = LinearGaussian(ndim_state=6, mapping=[0, 2, 4, 5],
                                   noise_covar=np.diag([1**2, 1**2, 3**2, 3**2]))
```

Defining the tracker components
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
With the state-space models defined, we proceed to build our tracking components

Filtering
*********
Since we have assumed Linear-Gaussian models, we will be using a Kalman Filter to perform
filtering of the underlying single-target densities. This is done by making use of the
:class:`~.KalmanPredictor` and :class:`~.KalmanUpdater` classes, which we define below:

```python
from stonesoup.predictor.kalman import KalmanPredictor
predictor = KalmanPredictor(transition_model)
```

```python
from stonesoup.updater.kalman import KalmanUpdater
updater = KalmanUpdater(measurement_model)
```

Note

For more information on the above classes and how they operate you can refer to the Stone
  Soup tutorial on
  `using the Kalman Filter <https://stonesoup.readthedocs.io/en/latest/auto_tutorials/01_KalmanFilterTutorial.html>`_.

Data Association
****************
We utilise a :class:`~.DistanceHypothesiser` to generate hypotheses between tracks and
measurements, where :class:`~.Mahalanobis` distance is used as a measure of quality:

```python
from stonesoup.hypothesiser.distance import DistanceHypothesiser
from stonesoup.measures import Mahalanobis
hypothesiser = DistanceHypothesiser(predictor, updater, Mahalanobis(), 10)
```

Continuing the :class:`~.GNNWith2DAssigment` class is used to perform fast joint data association,
based on the Global Nearest Neighbour (GNN) algorithm:

```python
from stonesoup.dataassociator.neighbour import GNNWith2DAssignment
data_associator = GNNWith2DAssignment(hypothesiser)
```

Note

For more information on the above classes and how they operate you can refer to the
  `Data Association - clutter <https://stonesoup.readthedocs.io/en/latest/auto_tutorials/05_DataAssociation-Clutter.html>`_
  and `Data Association - Multi-Target Tracking <https://stonesoup.readthedocs.io/en/latest/auto_tutorials/06_DataAssociation-MultiTargetTutorial.html>`_
  tutorials.

Track Initiation
****************
For initialising tracks we will use a :class:`~.MultiMeasurementInitiator`, which allows our
tracker to tentatively initiate tracks from unassociated measurements, and hold them within the
initiator until they have survived for at least 10 frames. We also define a
:class:`~.UpdateTimeStepsDeleter` deleter to be used by the initiator to delete tentative tracks
that have not been associated to a measurement in the last 3 frames.

```python
from stonesoup.types.state import GaussianState
from stonesoup.types.array import CovarianceMatrix, StateVector
from stonesoup.initiator.simple import MultiMeasurementInitiator
from stonesoup.deleter.time import UpdateTimeStepsDeleter
prior_state = GaussianState(StateVector(np.zeros((6,1))),
                            CovarianceMatrix(np.diag([100**2, 30**2, 100**2, 30**2, 100**2, 100**2])))
deleter_init = UpdateTimeStepsDeleter(time_steps_since_update=3)
initiator = MultiMeasurementInitiator(prior_state, measurement_model, deleter_init,
                                      data_associator, updater, min_points=10)
```

Track Deletion
**************
For confirmed tracks we used again a :class:`~.UpdateTimeStepsDeleter`, but this time configured
to delete tracks after they have not bee associated to a measurement in the last 15 frames.

```python
deleter = UpdateTimeStepsDeleter(time_steps_since_update=15)
```

Note

For more information on the above classes and how they operate you can refer to the Stone
  `Initiators & Deleters <https://stonesoup.readthedocs.io/en/latest/auto_tutorials/09_Initiators_&_Deleters.html>`_
  tutorial.

Building the tracker
~~~~~~~~~~~~~~~~~~~~
Now that we have defined all our tracker components we proceed to build our multi-target tracker:

```python
from stonesoup.tracker.simple import MultiTargetTracker
tracker = MultiTargetTracker(
    initiator=initiator,
    deleter=deleter,
    detector=detector,
    data_associator=data_associator,
    updater=updater,
)
```

Running the tracker
~~~~~~~~~~~~~~~~~~~

```python
def draw_tracks(image, tracks, show_history=True, show_class=True, show_score=True):
    """ Draw tracks on an image

    Parameters
    ----------
    image: :class:`PIL.Image`
        Image on which to draw the tracks
    detections: : set of :class:`~.Tracks`
        A set of tracks generated by our :class:`~.MultiTargetTracker`
    show_history: bool
        Whether to draw the trajectory of the track. Default is ``True``
    show_class: bool
        Whether to draw the class of the object. Default is ``True``
    show_score: bool
        Whether to draw the score of the object. Default is ``True``

    Returns
    -------
    : :class:`PIL.Image`
        Image with tracks drawn

    """
    draw = ImageDraw.Draw(image)
    for track in tracks:
        bboxes = np.array([np.array(state.state_vector[[0, 2, 4, 5]]).reshape(4)
                           for state in track.states])
        x0, y0, w, h = bboxes[-1]
        x1 = x0 + w
        y1 = y0 + h
        draw.rectangle([x0, y0, x1, y1], outline=(255, 0, 0), width=2)

        if show_history:
            pts = [(box[0] + box[2] / 2, box[1] + box[3] / 2) for box in bboxes]
            draw.line(pts, fill=(255, 0, 0), width=2)

        class_ = track.metadata['class']['name']
        score = round(float(track.metadata['score']), 2)
        if show_class and show_score:
            draw.text((x0, y1 + 2), '{}:{}'.format(class_, score), fill=(255, 0, 0))
        elif show_class:
            draw.text((x0, y1 + 2), '{}'.format(class_), fill=(255, 0, 0))
        elif show_score:
            draw.text((x0, y1 + 2), '{}'.format(score), fill=(255, 0, 0))
    return image

fig3, ax3 = plt.subplots(num="MultiTargetTracker output")
fig3.tight_layout()
artists3 = []
print("Running MultiTargetTracker example... Be patient...")
for timestamp, tracks in tracker:
    if not (len(artists3) + 1) % 10:
        print("Frame: {}/{}".format(len(artists3) + 1, num_frames))

    # Read the detections
    detections = detector.detections

    # Read frame
    frame = frame_reader.frame
    pixels = copy(frame.pixels)

    # Plot output
    image = Image.fromarray(pixels)
    image = draw_detections(image, detections)
    image = draw_tracks(image, tracks)
    ax3.axes.xaxis.set_visible(False)
    ax3.axes.yaxis.set_visible(False)
    fig3.tight_layout()
    artist = ax3.imshow(image, animated=True)
    artists3.append([artist])
ani3 = animation.ArtistAnimation(fig3, artists3, interval=20, blit=True, repeat_delay=200)
```


```python
import kenlm
import cPickle as pickle
import numpy as np
import copy
```

```python
beam_size = 10
# templates_dump_location = './eo_with_surf_forms_templates.p'
templates_dump_location = './ar_without_surf_forms_templates.p'
# Whether to sample from sentences with surface form tuples and property placeholders OR with pure text.
# summaries_type = 'summary_with_surf_forms_and_types'
summaries_type = 'original_summary'
```

### Loading trained $n$-gram model along with the original dataset file

```python
model = kenlm.LanguageModel('ar_without_surf_forms.klm')
dataset_file_location = '../Datasets/ar/Dataset/with-Surface-Forms/splitDataset_with_targets.p'
with open(dataset_file_location, 'rb') as f:
    dataset = pickle.load(f)
```

### Building vocabulary

```python
vocab_len = 0
token2id = {}
id2token = {}

for i in range(0, len(dataset['train'][summaries_type])):
    tempSummary = dataset['train'][summaries_type][i].encode('utf-8').split()
    for token in tempSummary:
        if token not in token2id:
            vocab_len += 1
            token2id[token] = vocab_len
            id2token[vocab_len] = token
vocab_len += 1
token2id['</s>'] = vocab_len
id2token[vocab_len] = '</s>'

# Make sure that sequences are absolved from start- and end-of-sequence tokens.
# These are not handled very well by this model.
assert('<start>' not in token2id)
assert('<end>' not in token2id)
```

### Beam-search decoding on the trained $n$-gram model
We initialise the beams with the $n$ most probable words given the `<s>` token (i.e. `<start>` token in KenLM Language Model Toolkit)

More information at: https://kheafield.com/code/kenlm/

```python
sentences = []
sentences_prob = []
candidates = []
num_active_beams = beam_size

beam_probabilities = np.zeros(vocab_len)
for token in token2id:
    tempCandidate = ' '.join([] + [token])
    beam_probabilities[token2id[token] - 1] = model.score(tempCandidate, eos = False)
indices = np.argsort(beam_probabilities)[-num_active_beams:]
print indices
for j in range(beam_size - 1, -1, -1):
    candidates.append([id2token[indices[j] + 1]])

while num_active_beams > 0:
    beam_probabilities = np.zeros(num_active_beams * vocab_len)
    beam_probabilities.fill(np.NINF)
    for s in range(0, num_active_beams):
        for token in token2id:
            tempCandidate = ' '.join(candidates[s] + [token])
            beam_probabilities[s * vocab_len + token2id[token] - 1] = model.score(tempCandidate, eos = False)

    indices = np.argsort(beam_probabilities)[-num_active_beams:]
    print indices
    cloned_candidates = copy.deepcopy(candidates)
    completed_beams_counter = 0
    candidates = []
    for j in range(num_active_beams - 1, -1, -1):

        candidates.append([])
        candidates[-1] = copy.deepcopy(cloned_candidates[indices[j] / vocab_len])

        candidates[-1] += [id2token[indices[j] % vocab_len + 1]]
        if id2token[indices[j] % vocab_len + 1] == '</s>':
            completed_beams_counter += 1
            sentences.append(candidates[-1])
            sentences_prob.append(beam_probabilities[indices[j]])
            candidates.pop(-1)
    num_active_beams -= completed_beams_counter
```

    [423458     23   1197    487   3478    639    421     10      7     43]
    [ 852805  864784    1078  848042    6411  423502 1694807     784  847022
      423553]
    [1271582      95 2545349 2541879 3812537 1693845 2964375  847080  437077
         120]
    [3813604    1706 3394202 3813713 3387768 2541328 1693846 1271165  847703
         121]
    [1270519  847997 2964238 2541725  861699     504 2117392  846968 1270518
      423585]
    [ 865303  856119  854492 3387682    2804 2964333 2117305  864152  423469
         786]
    [3811141 2975390 3396874 2541140 1700102 2120100 1270763  846928  423601
         386]
    [1694222 1277951    3619  847304   19845    6889    2628    4774     787
       10919]
    [3811181 3398849 2965001 2118083 2540804 1694624 1278554  851664  424247
         788]
    [  10920  436005 3387682  423509     400   19710 2134529      50 1271165
      847706]
    [2565451 2129236 3387800 2964312    6168      50 1693964  431033 1270387
      859251]
    [3811259 3387800 2964274 1698610 2555194 2117423 1270438  863225  423558
         128]
    [3811192 3387733 2964275 2551454 2540753 1693897 1270439  847046  437899
          61]
    [2599823 2969594 3387734 2964275 2117423 1693898  846979  846917      62]
    [ 267174   53271   34674   27153   17004  428840 1270438  423521]
    [2964343  224861 1270507 1693966  854444   23109   24350  423521]
    [2964344  847048  423589  151129   60389 2117426 1693967    7256]
    [2964274     130 1693966  423520 1270507 2540885  846979 2117426]
    [2964345 2540885 2117426      61 1693968  846979 1270508  423591]
    [ 423520 2964346 2540815 2118086 1693968 1270510  847050     133]
    [2964345 2540888 2117427 1694628 1270510  847052  423592     134]
    [2964346 2964212 1693969 2117428 1270511 1270376  423593  423458]
    [1693970 1271047  847052  846917  423458]
    [1270376  423915  423458]
    [423458]

### Purely-greedy decoding on the trained $n$-gram model

```python
# sentences = [[]]
# sentences_prob = [np.NINF]
# selectedToken = ''

# while selectedToken != '</s>':
#     max_prob = np.NINF
#     for token in token2id:
#         tempCandidate = ' '.join(sentences[0] + [token])
#         sentences_prob[-1] = tempScore
#         tempScore = model.score(tempCandidate, eos = False)
#         if tempScore > max_prob:
#             selectedToken = token
#             max_prob = tempScore
#     sentences[0].append(selectedToken)

#     print(sentences[0])
```

```python
for s in sentences:
    print s
```

    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x84\xd9\x88\xd9\x83', '\xd9\x81\xd9\x8a', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd8\xaa\xd9\x84', '\xd8\xa3\xd8\xa8\xd9\x8a\xd8\xb6', '\xd9\x81\xd9\x8a', '\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xa7\xd9\x84\xd8\xb1\xd9\x82\xd8\xa9.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd9\x85\xd8\xb1\xd9\x83\xd8\xb2', '\xd8\xa7\xd9\x84\xd8\xb1\xd9\x82\xd8\xa9', '\xd9\x81\xd9\x8a', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd9\x85\xd8\xb1\xd9\x83\xd8\xb2', '\xd8\xa7\xd9\x84\xd8\xb1\xd9\x82\xd8\xa9', '\xd9\x81\xd9\x8a', '\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xa7\xd9\x84\xd8\xb1\xd9\x82\xd8\xa9.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd8\xa5\xd8\xaf\xd8\xa7\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa7\xd9\x8b', '\xd9\x84\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xad\xd9\x84\xd8\xa8', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd9\x85\xd9\x86\xd8\xa8\xd8\xac', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd9\x85\xd8\xb1\xd9\x83\xd8\xb2', '\xd9\x85\xd9\x86\xd8\xa8\xd8\xac\xd8\x8c', '\xd8\xa8\xd9\x84\xd8\xba', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x87\xd8\xa7', '195', '\xd9\x86\xd8\xb3\xd9\x85\xd8\xa9', '\xd8\xad\xd8\xb3\xd8\xa8', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xa7\xd9\x84\xd9\x8a\xd9\x85\xd9\x86', '\xd9\x84\xd8\xb9\xd8\xa7\xd9\x85', '2004.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd8\xa5\xd8\xaf\xd8\xa7\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa7\xd9\x8b', '\xd9\x84\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xad\xd9\x84\xd8\xa8', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd9\x85\xd9\x86\xd8\xa8\xd8\xac', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd9\x85\xd8\xb1\xd9\x83\xd8\xb2', '\xd9\x85\xd9\x86\xd8\xa8\xd8\xac\xd8\x8c', '\xd8\xa8\xd9\x84\xd8\xba', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x87\xd8\xa7', '282', '\xd9\x86\xd8\xb3\xd9\x85\xd8\xa9', '\xd8\xad\xd8\xb3\xd8\xa8', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xa7\xd9\x84\xd9\x8a\xd9\x85\xd9\x86', '\xd9\x84\xd8\xb9\xd8\xa7\xd9\x85', '2004.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd8\xa5\xd8\xaf\xd8\xa7\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa7\xd9\x8b', '\xd9\x84\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xad\xd9\x84\xd8\xa8', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd9\x85\xd9\x86\xd8\xa8\xd8\xac', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd9\x85\xd8\xb1\xd9\x83\xd8\xb2', '\xd9\x85\xd9\x86\xd8\xa8\xd8\xac\xd8\x8c', '\xd8\xa8\xd9\x84\xd8\xba', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x87\xd8\xa7', '846', '\xd9\x86\xd8\xb3\xd9\x85\xd8\xa9', '\xd8\xad\xd8\xb3\xd8\xa8', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xa7\xd9\x84\xd9\x8a\xd9\x85\xd9\x86', '\xd9\x84\xd8\xb9\xd8\xa7\xd9\x85', '2004.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd8\xa5\xd8\xaf\xd8\xa7\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa7\xd9\x8b', '\xd9\x84\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xad\xd9\x84\xd8\xa8', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd8\xac\xd8\xa8\xd9\x84', '\xd8\xb3\xd9\x85\xd8\xb9\xd8\xa7\xd9\x86', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd8\xaa\xd9\x84', '\xd8\xa7\xd9\x84\xd8\xb6\xd9\x85\xd8\xa7\xd9\x86\xd8\x8c', '\xd8\xa8\xd9\x84\xd8\xba', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x87\xd8\xa7', '332', '\xd9\x86\xd8\xb3\xd9\x85\xd8\xa9', '\xd8\xad\xd8\xb3\xd8\xa8', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xa7\xd9\x84\xd9\x8a\xd9\x85\xd9\x86', '\xd9\x84\xd8\xb9\xd8\xa7\xd9\x85', '2004.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd8\xa5\xd8\xaf\xd8\xa7\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa7\xd9\x8b', '\xd9\x84\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xad\xd9\x84\xd8\xa8', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd8\xac\xd8\xa8\xd9\x84', '\xd8\xb3\xd9\x85\xd8\xb9\xd8\xa7\xd9\x86', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd8\xaa\xd9\x84', '\xd8\xa7\xd9\x84\xd8\xb6\xd9\x85\xd8\xa7\xd9\x86\xd8\x8c', '\xd8\xa8\xd9\x84\xd8\xba', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x87\xd8\xa7', '295', '\xd9\x86\xd8\xb3\xd9\x85\xd8\xa9', '\xd8\xad\xd8\xb3\xd8\xa8', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xa7\xd9\x84\xd9\x8a\xd9\x85\xd9\x86', '\xd9\x84\xd8\xb9\xd8\xa7\xd9\x85', '2004.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd8\xa5\xd8\xaf\xd8\xa7\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa7\xd9\x8b', '\xd9\x84\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xad\xd9\x84\xd8\xa8', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd8\xb9\xd9\x8a\xd9\x86', '\xd8\xa7\xd9\x84\xd8\xb9\xd8\xb1\xd8\xa8', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd9\x85\xd8\xb1\xd9\x83\xd8\xb2', '\xd8\xb9\xd9\x8a\xd9\x86', '\xd8\xa7\xd9\x84\xd8\xb9\xd8\xb1\xd8\xa8\xd8\x8c', '\xd8\xa8\xd9\x84\xd8\xba', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x87\xd8\xa7', '76', '\xd9\x86\xd8\xb3\xd9\x85\xd8\xa9', '\xd8\xad\xd8\xb3\xd8\xa8', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xa7\xd9\x84\xd9\x8a\xd9\x85\xd9\x86', '\xd9\x84\xd8\xb9\xd8\xa7\xd9\x85', '2004.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd8\xa5\xd8\xaf\xd8\xa7\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa7\xd9\x8b', '\xd9\x84\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xad\xd9\x84\xd8\xa8', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd8\xb9\xd9\x8a\xd9\x86', '\xd8\xa7\xd9\x84\xd8\xb9\xd8\xb1\xd8\xa8', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd9\x85\xd8\xb1\xd9\x83\xd8\xb2', '\xd8\xb9\xd9\x8a\xd9\x86', '\xd8\xa7\xd9\x84\xd8\xb9\xd8\xb1\xd8\xa8\xd8\x8c', '\xd8\xa8\xd9\x84\xd8\xba', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x87\xd8\xa7', '649', '\xd9\x86\xd8\xb3\xd9\x85\xd8\xa9', '\xd8\xad\xd8\xb3\xd8\xa8', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xa7\xd9\x84\xd9\x8a\xd9\x85\xd9\x86', '\xd9\x84\xd8\xb9\xd8\xa7\xd9\x85', '2004.', '</s>']
    ['\xd9\x82\xd8\xb1\xd9\x8a\xd8\xa9', '\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa9', '\xd8\xaa\xd8\xaa\xd8\xa8\xd8\xb9', '\xd8\xa5\xd8\xaf\xd8\xa7\xd8\xb1\xd9\x8a\xd9\x91\xd8\xa7\xd9\x8b', '\xd9\x84\xd9\x85\xd8\xad\xd8\xa7\xd9\x81\xd8\xb8\xd8\xa9', '\xd8\xad\xd9\x84\xd8\xa8', '\xd9\x85\xd9\x86\xd8\xb7\xd9\x82\xd8\xa9', '\xd8\xb9\xd9\x8a\xd9\x86', '\xd8\xa7\xd9\x84\xd8\xb9\xd8\xb1\xd8\xa8', '\xd9\x86\xd8\xa7\xd8\xad\xd9\x8a\xd8\xa9', '\xd9\x85\xd8\xb1\xd9\x83\xd8\xb2', '\xd8\xb9\xd9\x8a\xd9\x86', '\xd8\xa7\xd9\x84\xd8\xb9\xd8\xb1\xd8\xa8\xd8\x8c', '\xd8\xa8\xd9\x84\xd8\xba', '\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x87\xd8\xa7', '1,142', '\xd9\x86\xd8\xb3\xd9\x85\xd8\xa9', '\xd8\xad\xd8\xb3\xd8\xa8', '\xd8\xa7\xd9\x84\xd8\xaa\xd8\xb9\xd8\xaf\xd8\xa7\xd8\xaf', '\xd8\xa7\xd9\x84\xd8\xb3\xd9\x83\xd8\xa7\xd9\x86\xd9\x8a', '\xd9\x84\xd8\xb9\xd8\xa7\xd9\x85', '2004', '.', '</s>']

```python
for s in range(0, len(sentences)):
    assert(sentences[s][-1] == '</s>')
    sentences[s] = ' '.join(sentences[s][:-1]).decode('utf-8')
    print sentences[s], sentences_prob[s]
```

    قرية سورية تتبع ناحية سلوك في منطقة تل أبيض في محافظة الرقة. -4.30183410645
    قرية سورية تتبع ناحية مركز الرقة في منطقة مركز الرقة في محافظة الرقة. -4.10472583771
    قرية سوريّة تتبع إداريّاً لمحافظة حلب منطقة منبج ناحية مركز منبج، بلغ تعداد سكانها 195 نسمة حسب تعداد اليمن لعام 2004. -6.04410171509
    قرية سوريّة تتبع إداريّاً لمحافظة حلب منطقة منبج ناحية مركز منبج، بلغ تعداد سكانها 282 نسمة حسب تعداد اليمن لعام 2004. -6.12608671188
    قرية سوريّة تتبع إداريّاً لمحافظة حلب منطقة منبج ناحية مركز منبج، بلغ تعداد سكانها 846 نسمة حسب تعداد اليمن لعام 2004. -6.34612083435
    قرية سوريّة تتبع إداريّاً لمحافظة حلب منطقة جبل سمعان ناحية تل الضمان، بلغ تعداد سكانها 332 نسمة حسب تعداد اليمن لعام 2004. -6.11217737198
    قرية سوريّة تتبع إداريّاً لمحافظة حلب منطقة جبل سمعان ناحية تل الضمان، بلغ تعداد سكانها 295 نسمة حسب تعداد اليمن لعام 2004. -6.14283323288
    قرية سوريّة تتبع إداريّاً لمحافظة حلب منطقة عين العرب ناحية مركز عين العرب، بلغ تعداد سكانها 76 نسمة حسب تعداد اليمن لعام 2004. -6.26259183884
    قرية سوريّة تتبع إداريّاً لمحافظة حلب منطقة عين العرب ناحية مركز عين العرب، بلغ تعداد سكانها 649 نسمة حسب تعداد اليمن لعام 2004. -6.38466739655
    قرية سوريّة تتبع إداريّاً لمحافظة حلب منطقة عين العرب ناحية مركز عين العرب، بلغ تعداد سكانها 1,142 نسمة حسب التعداد السكاني لعام 2004 . -6.36081552505

### Computing probabilities
KenLM scores are $log_{10}$ probabilities; we compute the actual ones (i.e. $0 \leq p_i \leq 1$).
We normalise as follows:
\begin{equation}
\widetilde{p_i} = \frac{p_i}{\sum_i p_i}
\end{equation}

```python
nominator = np.power(10, np.asarray(sentences_prob))
denominator = np.power(10, np.asarray(sentences_prob)).sum()
```

```python
prob_distribution = nominator / denominator
```

```python
print prob_distribution
```

    [ 0.37392506  0.58869839  0.00676888  0.00560444  0.00337674  0.00578684
      0.00539244  0.00409286  0.00308995  0.0032644 ]

### Saving everything in a dictionary
```python
{'sentences': type(list), 'prob_distribution': type(numpy.ndarray)}
```

```python
with open(templates_dump_location, 'wb') as f:
    pickle.dump({'sentences': sentences, 'prob_distribution': prob_distribution}, f)
```


Table of Contents

1  Gated CNN

1.1  GCNN 1D Time Series

1.2  GCNN 1D Residuals

```python

#####
###  Inicio de refatoracao usando principios de design patterns.
# TODO: pesquisar mais e mais e mais a fundo o design do Tensorflow pra seguir de perto.

from enum import Enum

class BlockTypes(Enum):
  GCNN2D = 'gcnn2d'
  SOFTMAX = 'softmax'
  CROSSENTROPY = 'loss-crossentropy'
  INPUT = 'signal_in'
  REDUCEMEAN = 'reducemean'

class ArchBlocks:

  def __add__(self, block):
    pass

class Architecture(ArchBlocks):
  '''
    aa
  '''
  def __init__(self, **kwargs):

    self.modules = {'0':[ArchBlocks.INPUT,0,0,[],[]]}

  def __add__(self, archblock1):
    pass
    #self.modules = archblock1

  def _block_specification(self,key):
    pass

class Stattus4NeuralNetAPI:

  def __init__(self, datatype, datasocket = False, **kwargs):
    '''
     Input

     datatype: Audio, Pressure, Flow.
     datasocket: if True tries to fetch data from a database and push output to database
     pre-configured through kwargs, if config is not provided then tries to read from datafeeder or dump to local file-system.

     Optional
     num_of_channels: number of channels for Audio data, if not given get only the first audio and discard the others.
    '''
    self.datatype = datatype
    self.datasocket = datasocket

  def setup_hyperparameters(self, architectures):
    '''
      Input

      architectures: Architecture object specifying archetypical structure to be investigated for its hyperparams.

      Return a dictionary of the given
    '''
    pass

#######
#######
 ####
  ###
   #

#  FUTURE: Name/Var Scope handler. This class/template must be used to implement logic of names
# according to the architecture block connection. E.G. if a block is just the same from the other in the
# sequence them it must be named as if is a deep part of the same architecture blocks. If it is a ramification
# of the given below but the same arch them he must be named with another tag.
# When duplicating some block to architectonic use, E.G. if you need to frame your data or feature, them each
# frame must have an coeherent name space block, like the today ordimatlicly done for repeated name space block,
# but this must be automaticly done for identical built blocks of the model, without the need to explicity code it.
# Names must be given as unique ID according to te input-output architecture connection, if it has the same IO from previous Block in the sequence,
# then each Input-ARchBlock-Output that is IDentical must have the same name with the appropriated tag for deepnes in the sequence of blocks
# Automatic update of the name scopes according to the architecture.
# Dont change Identical sequential IO Name spaces
```

```python
## Unused building-blocks, will add in the API in the future, after a throughful refatoration.
#

  self.architectures.gcnn1d = self._gcnn1d
  self.architectures.residualgcnn1d = self._residualgcnn1d
  def _gcnn1d(self, **kwargs):

    try:

      channels_out = kwargs['channels_out']
      filter_size = kwargs['filter_size']

    except Exception:

      sys.exit('Parameters Not Defined Error')

      signal_in = self(**kwargs)

    # postfix = self.get_namepostfix('gccn1d',**kwargs)
      with self.graph.as_default():
        with tf.variable_scope('gccn1d'):#+postfix):

          with self.graph.device(_dev_selector(arg1='foo')):
            conv_linear = tf.keras.layers.Conv1D( channels_out, filter_size, padding='causal', name='conv_linear', use_bias=True)(signal_in)

          with self.graph.device(_dev_selector(arg1='foo')):
            conv_gate = tf.sigmoid(tf.keras.layers.Conv1D( channels_out, filter_size, padding='causal', name='conv', use_bias=True )(signal_in),name='conv_sigmoid')

          with self.graph.device(_dev_selector(arg1='foo')):
            gated_convolutions = tf.multiply(conv_linear,conv_gate,name='gated_convolutions')

  def _residualgcnn1d(self, **kwargs):

    try:

      channels_out = kwargs['channels_out']
      filter_size = kwargs['filter_size']

    except KeyError:

      sys.exit('Parameters Not Defined Error')

    signal_in = self(**kwargs)

      ####
      ## Keras convolutions. Classes, so dont behave like functions but outputs Tensor, use its functions to query the variables filters and bias

        # postfix = self.get_namepostfix('gccn1d',**kwargs)
    with self.graph.as_default():
      with tf.variable_scope('residualgccn1d'):#+postfix):

        with self.graph.device(_dev_selector(arg1='foo')):
          conv_linear = tf.keras.layers.Conv1D( channels_out, filter_size, padding='causal', name='conv_linear', use_bias=True)(signal_in)
        with self.graph.device(_dev_selector(arg1='foo')):
          conv_gate = tf.sigmoid(tf.keras.layers.Conv1D( channels_out, filter_size, padding='causal', name='conv', use_bias=True )(signal_in),name='conv_sigmoid')
        with self.graph.device(_dev_selector(arg1='foo')):
          gated_convolutions = tf.multiply(conv_linear,conv_gate,name='gated_convolutions')

      # Input channels must be the same size of the convolution channels output (ie number of filters applied)
      with self.graph.device(_dev_selector(arg1='foo')):
        residual = tf.add(gated_convolutions,signal_in,name='residual')

  def _save_trainable_vars(self, blockname):
    with self.graph.as_default():

      saver = tf.train.Saver(var_list=self.graph.get_collection('trainable_variables') )
      tf.global_variables_initializer()

      sess = tf.Session(graph=self.graph)
      sess.run(self.graph.get_operations()[-1])
      saver.save(sess, os.getcwd()+'/'+blockname)
      #saver.export_meta_graph(filename=blockname+'.constructor', collection_list='trainable_variables', export_scope=None, strip_default_attrs=False)
      self.arch_blocks[blockname] = blockname

  def define_block(self,blockname):

    self._save_trainable_vars(blockname)

    graph = tf.Graph()
    with graph.as_default():
      sess = tf.Session(graph=graph)

      new_saver = tf.train.import_meta_graph(os.getcwd()+'/'+self.arch_blocks[blockname]+'.meta', import_scope = blockname)
      new_saver.restore(sess, os.getcwd()+'/'+self.arch_blocks[blockname])
      ####
      ## Redefinir signal_in dos blocos utilizados para construir este bloco através do dict namescopo
      ## para que run_cgraph consiga achar inputs
      #

      ####
      ## Signalin handling provisorio, fazer um modulo apenas para lidar com signal
      #
      self.graph = graph
      self.signal_in = []
      self.signal_in.append(self.graph.get_tensor_by_name(blockname+'/signal_in:0'))

  # Build block from name in the top level of the blocks, this is a previous version that take names inside namescope dict and build new block
  def from_block(self,archblockname,tag = ""):

    with self.graph.as_default():
      sess = tf.Session(graph=self.graph)

      new_saver = tf.train.import_meta_graph(os.getcwd()+'/'+self.arch_blocks[archblockname]+'.meta', import_scope = archblockname+tag)
      new_saver.restore(sess, os.getcwd()+'/'+self.arch_blocks[archblockname])

      self.signal_in.append(self.graph.get_tensor_by_name(archblockname+tag+'/signal_in:0'))
      self.num_input -= 1

```

##  Gated CNN
  Gated CNN is a doubled CNN in whom one of the convoluted signals does the role of opening/closing the network, giving an **Attention Mechanism** to the convolution, for being activated by a sigmoid.
  It gives non-vanishing gradient, since the multiplication rule for the derivative applies, also, applies gradient to the linear convoluted part.

### GCNN 1D Time Series
  In time series version, since the desirable learning is based on **past** events, or you cannot uphold the assumption that you have acess to future data, have to make sure that the convolution is **causal**, that is
\begin{align}
y_{n}= a_{i}x_{n-i}=a_{n-j}x_{j}
\end{align}
  Giving at last, if the filter has length k, k-1 zero padding to the input x.

```python
#build graph
graph = build_graph( (None,l[0],channels), arch = 'gcnn1d',print_ops = True, new_graph=True, show_cgraph = True)
'''
obs: if you have a graph sometimes is necessary to run more than one time wit reset_default_graph to get a new graph
'''
```

### GCNN 1D Residuals

```python
#build graph
graph = build_graph( (None,l[0],channels), arch = 'residual gcnn1d',print_ops = True,new_graph=True, show_cgraph = True)
```

# Hyper Param Protocol (hpp)

Two interfaces implementing the following:

A - Space x Selection Algorithm x Architecture

- Implementations of this interface specifies the search space for the Deep Learning model, including space size i.e. the number of points and space complexity that is objects to data dict (Variability)/structure/number of sources/Unique Ids (e.g. location). The Space Complex objects should be implement through his own interface.

B - Space Complex

- Central to space complexity are the specifications of the already given above. Once one have implemented the specification than can implement the Interface itself.

```python
#   HPP Definitions. First DISS  implementation by the already made architecture, brute force selection Algol,
# and Space == Space Size.
# TODO - refatorar.
# TODO - full implementation

class Hpp:
  '''
   Hyperparam tuning protocol.

   Input
   - space: Space object giving the size and the complexity of the space
   - selection_algorithm: function running algorithm that search through space using specific evaluation (also implemented in the algorithm)
   - architecture: Architecture object specifying archetipical architecture to be tuned by searching in space.
  '''

  def __init__(self,space = {},selection_algorithm = lambda x: 'foo',architecture='gcnn2d'):

    self.space_size = space.get_size()
    self.data_structure = space.get_complex().data.get_structure()
    self.source_id = space.get_complex().get_ids()
    self.data_dict = space.get_complex().data.get_dict()
    self.algorithm = selection_algorithm
    self.architecture = architecture

  def _parse_config(config_list):
    '''
      Receives a config and build from dict to perform  examples:
      config0 = [ {"0_frames": lambda : 4 1 if 10 == 1 else 0,"0_0": lambda :(10,30,40,2), "0_1": lambda :(10,2,2,8), "0_2": lambda :(10,2,1,16), "0_3": lambda :(10,4,1,32), "0_4": lambda :(10,8,1,64), "0_5": lambda :(10,16,1,128), "0_6": lambda :(10,13,1,256)},{"1_labels": lambda :2},{"2_null": lambda :'a'},{"3_labels": lambda :2,"3_learningrate": lambda :0.005}]
      config1 = [ {"0_frames":4,"0_0":(10,30,40,2)}, {"1_1":(10,2,2,8), "1_2":(10,2,1,16), "1_3":(10,4,1,32), "1_4":(10,8,1,64), "1_5":(10,8,1,64)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config2 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8), "0_2":(10,2,1,16), "0_3":(10,4,1,32), "0_4":(10,8,1,64) },{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config3 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8), "0_2":(10,2,1,16), "0_3":(10,4,1,32) },{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config4 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8), "0_2":(10,2,1,16)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config5 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]

      config6 = [{"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8), "0_2":(10,6,1,16), "0_3":(10,6,1,32), "0_4":(10,6,1,64), "0_5":(10,10,1,128), "0_6":(10,16,1,256)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config7 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8), "0_2":(10,6,1,16), "0_3":(10,6,1,32), "0_4":(10,6,1,64), "0_5":(10,10,1,128)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config8 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8), "0_2":(10,6,1,16), "0_3":(10,6,1,32), "0_4":(10,6,1,64)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config9 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8), "0_2":(10,6,1,16), "0_3":(10,6,1,32)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config10 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8), "0_2":(10,6,1,16)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
      config11 = [ {"0_frames":4,"0_0":(10,30,40,2), "0_1":(10,2,2,8)},{"1_labels":2},{"2_null":a},{"3_labels":2,"3_learningrate":0.005}]
    '''
    overall_config = []
    overall_attr = []
    overall_deepness = []
    for block_config in config_list:
      layer_configs = []
      layer_attr = []
      layer_deep = 0
      # attributes
      bc = block_config.items()
      for item in bc:
        if item[0].split('_')[1].isalnum():
          layer_configs.append(item)
          layer_deep += 1
        else:
          layer_attr.append(item)

      layer_configs.sort()
      layer_attr.sort()

      overall_deepness.append(layer_deep)
      overall_config.append( dict(layer_configs) )
      overall_attr.append( dict(layer_attr) )

    return overall_attr,overall_config,overall_deepness

  def _is_vec_embed():
    pass

  def run(self):
    '''
    try:
      assert type(tf.get_default_graph()) == type(tf.Graph())
      graph = tf.get_default_graph()
    except AssertionError:
      graph = tf.Graph()
    '''

    tf.reset_default_graph()
    graph = tf.Graph()
    s = 0
    for config in range(self.algorithm.size):

      s += 1
      layer_config_list = self.algorithm()
      overall_attr,overall_config,overall_deepness = _parse_config(layer_config_list)

      with tf.variable_scope('Config {}'.format(s)):

        #TODO Assert that it is a block 0 and its attributes (for this case only nframe)

        blockattr = overall_attr.pop(0)
        nframe = blockattr.values()[0]
        config = overall_config.pop(0)

        for blocks in range(len(self.architecture)):

          graph = build_graph_module(graph, scope_tensor_name=op.name, arch = self.architecture[blocks], print_ops = True, name_scope = True, show_cgraph = True, filter_size=(k[1],k[2]), channels_out = k[3], deepness = '_d1',num_labels=2,learning_rate=0.005)
          op = graph.get_operations()[-1]

          try:
            blockattr = overall_attr[0]
            if blockattr.keys()[0].split('_')[0] == blocks:
              blockattr = overall_attr.pop(0)
              for attr,v in blockattr:
                if attr.split('_')[1] == 'frames':
                  nframes

          except:
            pass

        for k in layer_config[2:]:

            '''
        names = []
for op in graph.get_operations():

  if op.name.split('/')[1] == 'transpose_1' and op.name.split('/')[0][0:7] == 'softmax':
    names.append(op.name)

graph = build_graph_module(graph, scope_tensor_name=names, arch = 'reducemean',print_ops = True, name_scope = True, show_cgraph = True, filter_size=(2,1), channels_out = 64, deepness = '',num_labels=4,learning_rate=0.005, verbose=True)
op = graph.get_operations()[-1]
graph = build_graph_module(graph, scope_tensor_name=op.name, arch = 'loss-crossentropy',print_ops = True, name_scope = True, show_cgraph = True, filter_size=(2,1), channels_out = 64, deepness = '',num_labels=4,learning_rate=0.1, batch=40, verbose=True)

    layer_config = self.algorithm()
    with tf.variable_scope('secondparam'):

      graph = build_graph( (layer_config[0][0],layer_config[0][1],layer_config[0][2],layer_config[0][3]), arch = self.architecture,print_ops = True,new_graph=False, show_cgraph = True, filter_size=(layer_config[1][1],layer_config[1][2]), channels_out = layer_config[1][3])
      op = graph.get_operations()[-1]
      for k in layer_config[2:]:
        graph = build_graph_module(graph, scope_tensor_name=op.name, arch = self.architecture, print_ops = True, name_scope = True, show_cgraph = True, filter_size=(k[1],k[2]), channels_out = k[3], deepness = '_d1',num_labels=2,learning_rate=0.005)
        op = graph.get_operations()[-1]

      '''

class Space:
  '''
   Space is built upon space size and complexity.

   Input:
   - size: an integer, the number of configurations to look for the model.
   - spacecomplex: Unique Ids, a hashable type or string specifing source e.g. location. A SigmaComplex object specifying meta-data, i.e. data configurations and
   possibly other relevant keys to hyper-param searching such features, diversity mensures and so on.
   SigmaComplex must be an interface to be implemented according to the data specification, possibly separating mutable from immutable characters.
  '''

  def __init__(self, size, spacecomplex):
    self.size = size
    self.sigmacomplex = spacecomplex

  def get_size(self):
    return self.size

  def get_complex(self):
    return self.sigmacomplex

  def get_ids(self):
    return  self.sigmacomplex.get_ids()

  def get_sourcenum(self):
    return self.sigmacomplex.data.sourcenum

  def get_dict(self):
    return self.sigmacomplex.data.get_dict()

  def get_structure(self):
    return self.sigmacomplex.data.get_structure()

class SigmaComplex:
  '''
      Unique Ids, a hashable type or string specifing source e.g. location. A SigmaComplex object specifying meta-data, i.e. data configurations and
     possibly other relevant keys to hyper-param searching such features, diversity mensures and so on.
     SigmaComplex must be an interface to be implemented according to the data specification, possibly separating mutable from immutable characters.

     Data specification and ids will correspond to the empirical collected data according to the geo-located point in a device that has known inner workings
     according to the laws of physics.
  '''
  def __init__(self, ids, data):
    self.ids = ids
    self.data = data

  def get_ids(self):
    return  self.ids

  def get_dict(self):
    return self.data.get_dict()

  def get_structure(self):
    return self.data.get_structure()

class DataMeta:
  '''
   Data MaTter specification according to device collector, its inner workings (laws of physics), and other relevant keys.
  '''
  def __init__(self):

    self.structure = "image"
    self.data_dict = { "labels" : ["cv","sv"] , "features" : ["spect","framed"], "channel_num": 1 }

  def get_dict(self):

    return self.data_dict

  def get_structure(self):

    return self.structure
```

```python
# Example of 'foo'

spacecomplex = SigmaComplex(["Chala-head-chala"] , DataMeta())
space = Space( 5, spacecomplex)
hyperparam = Hpp(space, lambda x: '飛輪功', "gcnn2d")
```

```python
#  Build search space data structure. To be added to SigmaComplex class/interface/template
#  "softmax","reducemean","loss-crossentropy"

config0 = [ {"0_frames":4,"0_0":(10,30,2,2), "0_1":(10,2,2,8), "0_2":(10,2,1,16), "0_3":(10,4,1,32), "0_4":(10,8,1,64), "0_5":(10,16,1,128), {"1_labels":2} ]
config1 = [ {"0_frames":4,"0_0":(10,30,2,2), "0_1":(10,2,2,8), "0_2":(10,2,1,16), "0_3":(10,4,1,32)} ]
config2 = [ {"0_frames":4,"0_0":(10,30,2,2), "0_1":(10,2,2,8) ]
config3 = [ {"0_frames":4,"0_0":(10,30,2,2), "0_1":(10,4,2,8), "0_2":(10,4,1,16), "0_3":(10,4,1,32), "0_4":(10,4,1,64), "0_5":(10,7,1,128), "0_6":(10,7,1,256), "0_7":(10,7,1,512)}, {"1_labels":2},  ]
search_space = ( config0,  config1)
```

```python
# hash( '''( ( {"0_sld":4},{"0_0":(10,30.0,40.0)}),({"1_sld":3},{"1_0":(10,30.0,40.0,40.0)} ) )''' )
# Example exhaustive search (河南科技大学)
# you can use yield if you want to
class search(object):
  def __init__(self, listconfig):
    self.size = len(listconfig)
    self.num = 0
    self.config = listconfig

  def __iter__(self):
    return self

       # Python 3 compatibility
  def __next__(self):
    return self.next()

  def next(self):
    if self.num < self.size:
      cur, self.num = self.config[self.num], self.num+1
      return cur
    else:
      raise StopIteration()
  def __call__(self):
    return self.next()

spacecomplex = SigmaComplex(["Chala-head-chala"] , DataMeta())
space = Space( 5, spacecomplex)
#TODO implementation of the Space->SigmaComplex->DataMeta()
hyperparam = Hpp(space, search( search_space ), ["gcnn2d","softmax","reducemean","loss-crossentropy"])
```

```python

###
## Danijar option to define scope with decorators
#

def doublewrap(function):
    """
    A decorator decorator, allowing to use the decorator to be used without
    parentheses if no arguments are provided. All arguments must be optional.
    """
    @functools.wraps(function)
    def decorator(*args, **kwargs):
        if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):
            return function(args[0])
        else:
            return lambda wrapee: function(wrapee, *args, **kwargs)
    return decorator

@doublewrap
def define_scope(function, scope=None, *args, **kwargs):
    """
    A decorator for functions that define TensorFlow operations. The wrapped
    function will only be executed once. Subsequent calls to it will directly
    return the result so that operations are added to the graph only once.
    The operations added by the function live within a tf.variable_scope(). If
    this decorator is used with arguments, they will be forwarded to the
    variable scope. The scope name defaults to the name of the wrapped
    function.
    """
    attribute = '_cache_' + function.__name__
    name = scope or function.__name__
    @property
    @functools.wraps(function)
    def decorator(self):
        if not hasattr(self, attribute):
            with tf.variable_scope(name, *args, **kwargs):
                setattr(self, attribute, function(self))
        return getattr(self, attribute)
    return decorator
#####
####
##
#
```

```python
def _calculate_lr_alpha(self, step):

  self.lalpha = np.abs( (self.lossval[step] + self.lossval[step-1] + self.lossval[step-2] + self.lossval[step-3])/2.0 -self.lossval[step-4] -self.lossval[step-2])
  print('l ',self.lalpha)
  self.alpha = (self.lossval[step] - self.lossval[step-4])/self.lalpha
  print('alpha ', self.alpha)
```

```python
                        if len(self.logvarloss) >= 2:

            if self.logvarloss[-1] - self.logvarloss[-2] < - 1.0 and self.logvarloss[-1] - self.logvarloss[-2] > - 2.5:

              self.optimizer._lr = self.optimizer._lr + self.optimizer._lr/3.0

            elif self.logvarloss[-1] - self.logvarloss[-2] >  1.0 and self.logvarloss[-1] - self.logvarloss[-2] <  2.5:

              self.optimizer._lr = self.optimizer._lr - self.optimizer._lr/3.0

            elif self.logvarloss[-1] - self.logvarloss[-2] < - 2.5:

              self.optimizer._lr = self.optimizer._lr + self.optimizer._lr/2.0

            elif self.logvarloss[-1] - self.logvarloss[-2] >  2.5:

              self.optimizer._lr = self.optimizer._lr - self.optimizer._lr/2.0

            elif self.logvarloss[-1] - self.logvarloss[-2] < 1.0 and self.logvarloss[-1] - self.logvarloss[-2] > - 1.0:

              self.optimizer._lr = self.optimizer._lr + self.optimizer._lr/2.0
```