# Using a NiftyNet neural network model in PyTorch

## NiftyNet

```python

```

[NiftyNet](https://niftynet.io/) is "an open source convolutional neural networks platform for medical image analysis and image-guided therapy" built on top of [TensorFlow](https://www.tensorflow.org/). It is probably the easiest way to get started with deep learning for medical image and is mostly been used for segmentation.

*Image segmentation using deep learning* were the 5 most common words in all full paper titles from **both** [MICCAI 2018](https://www.miccai2018.org/en/) and [MIDL 2019](https://2019.midl.io/).

"Image segmentation using deep learning", guess this is the hottest topic in MIDL #MIDL2019 @midl_conference pic.twitter.com/64smdMjnxY
— Hua Ma (@forever_pippo) July 8, 2019

Just like #miccai2018! pic.twitter.com/3ZTHxj9iPT
— Julia Schnabel (@ja_schnabel) July 8, 2019

This is probably due to the great success that convolutional neural networks (CNNs) have achieved in this field in the past years.

## HighRes3DNet

HighRes3DNet is a residual convolutional neural network designed to have a large receptive field and preserve a high resolution using a relatively small number of parameters. It was presented in 2017 by Li et al. at IPMI: [*On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task*](https://arxiv.org/abs/1707.01992).

The authors used [NiftyNet](https://niftynet.io/) to train a model based on this architecture to perform [brain parcellation](https://ieeexplore.ieee.org/document/7086081?arnumber=7086081) from $T_1$-weighted MR images using the [ADNI dataset](http://adni.loni.usc.edu/).

This is an qualitative result the paper:

The code of the architecture is on [NiftyNet's GitHub repository](https://github.com/NifTK/NiftyNet/blob/dev/niftynet/network/highres3dnet.py) and the authors have have uploaded thre parameters and configuration file to the [Model Zoo](https://github.com/NifTK/NiftyNetModelZoo/tree/5-reorganising-with-lfs/highres3dnet_brain_parcellation). After reading the paper and the code, it is relatively straightforward to implement the architecture using PyTorch.

In this notebook we will:

1. [Extract the parameters from a TensorFlow checkpoint](#TensorFlow-world)
2. [Transform them to PyTorch](#PyTorch-world)
3. [Apply the model to some test data](#Run-inference)

## Setup

### Install and import libraries

```python
# %%capture
!pip install -r requirements.txt
```

    Requirement already satisfied: tensorflow-gpu==1.10 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.10.0)
    Requirement already satisfied: niftynet in /home/fernando/git/NiftyNet (from -r requirements.txt (line 2)) (0+untagged.1.gb6150f9.dirty)
    Requirement already satisfied: scikit-image in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (0.15.0)
    Requirement already satisfied: matplotlib in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (3.1.0)
    Requirement already satisfied: seaborn in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (0.9.0)
    Requirement already satisfied: pandas in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (0.24.2)
    Requirement already satisfied: nibabel in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from -r requirements.txt (line 7)) (2.4.1)
    Requirement already satisfied: torch in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (1.1.0)
    Requirement already satisfied: highresnet in /home/fernando/git/highresnet (from -r requirements.txt (line 9)) (0.2.0)
    Requirement already satisfied: setuptools<=39.1.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (39.1.0)
    Requirement already satisfied: protobuf>=3.6.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (3.8.0)
    Requirement already satisfied: astor>=0.6.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (0.8.0)
    Requirement already satisfied: numpy<=1.14.5,>=1.13.3 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (1.14.5)
    Requirement already satisfied: wheel>=0.26 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (0.33.4)
    Requirement already satisfied: six>=1.10.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (1.12.0)
    Requirement already satisfied: termcolor>=1.1.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (1.1.0)
    Requirement already satisfied: grpcio>=1.8.6 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (1.21.1)
    Requirement already satisfied: tensorboard<1.11.0,>=1.10.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (1.10.0)
    Requirement already satisfied: gast>=0.2.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (0.2.2)
    Requirement already satisfied: absl-py>=0.1.6 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorflow-gpu==1.10->-r requirements.txt (line 1)) (0.7.1)
    Requirement already satisfied: scipy>=0.18 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from niftynet->-r requirements.txt (line 2)) (1.3.0)
    Requirement already satisfied: configparser in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from niftynet->-r requirements.txt (line 2)) (3.7.4)
    Requirement already satisfied: pillow in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from niftynet->-r requirements.txt (line 2)) (6.0.0)
    Requirement already satisfied: blinker in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from niftynet->-r requirements.txt (line 2)) (1.4)
    Requirement already satisfied: packaging in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from niftynet->-r requirements.txt (line 2)) (19.0)
    Requirement already satisfied: imageio>=2.0.1 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from scikit-image->-r requirements.txt (line 3)) (2.5.0)
    Requirement already satisfied: PyWavelets>=0.4.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from scikit-image->-r requirements.txt (line 3)) (1.0.3)
    Requirement already satisfied: networkx>=2.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from scikit-image->-r requirements.txt (line 3)) (2.3)
    Requirement already satisfied: cycler>=0.10 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 4)) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.1 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 4)) (2.8.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 4)) (1.1.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 4)) (2.4.0)
    Requirement already satisfied: pytz>=2011k in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from pandas->-r requirements.txt (line 6)) (2019.1)
    Requirement already satisfied: markdown>=2.6.8 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorboard<1.11.0,>=1.10.0->tensorflow-gpu==1.10->-r requirements.txt (line 1)) (3.1.1)
    Requirement already satisfied: werkzeug>=0.11.10 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from tensorboard<1.11.0,>=1.10.0->tensorflow-gpu==1.10->-r requirements.txt (line 1)) (0.15.4)
    Requirement already satisfied: decorator>=4.3.0 in /home/fernando/miniconda3/envs/nn/lib/python3.6/site-packages (from networkx>=2.0->scikit-image->-r requirements.txt (line 3)) (4.4.0)

```python
%matplotlib inline
%config InlineBackend.figure_format='retina'

import tempfile
from pathlib import Path
from configparser import ConfigParser

import numpy as np
import pandas as pd

import torch
from torchsummary import summary

from niftynet.io.image_reader import ImageReader
from niftynet.engine.sampler_grid_v2 import GridSampler
from niftynet.engine.windows_aggregator_grid import GridSamplesAggregator
from niftynet.layer.pad import PadLayer
from niftynet.layer.binary_masking import BinaryMaskingLayer
from niftynet.layer.histogram_normalisation import HistogramNormalisationLayer
from niftynet.layer.mean_variance_normalisation import MeanVarNormalisationLayer

from highresnet import HighRes3DNet

import tf2pt
import utils
import visualization
```

    INFO:tensorflow:TensorFlow version 1.10.0
    INFO:tensorflow:Available Image Loaders:
    ['nibabel', 'opencv', 'skimage', 'pillow', 'simpleitk', 'dummy'].

### Download NiftyNet data

We can use [NiftyNet's `net_download`](https://github.com/NifTK/NiftyNetModelZoo/tree/5-reorganising-with-lfs/highres3dnet_brain_parcellation#downloading-model-zoo-files) to get all we need from the [Model Zoo](https://github.com/NifTK/NiftyNetModelZoo/tree/5-reorganising-with-lfs):

```python
try:
    import google.colab.data_table
except ModuleNotFoundError:
    print("We're not on Google Colab :(")
```

    We're not on Google Colab :(

```python
# %%capture
!net_download highres3dnet_brain_parcellation_model_zoo
```

    Accessing: https://github.com/NifTK/NiftyNetModelZoo
    highres3dnet_brain_parcellation_model_zoo: OK.
    Already downloaded. Use the -r option to download again.

```python
niftynet_dir = Path('~/niftynet').expanduser()
utils.list_files(niftynet_dir)
```

    niftynet/
        extensions/
            __init__.py
            network/
                __init__.py
            highres3dnet_brain_parcellation/
                __init__.py
                highres3dnet_config_eval.ini
        models/
            highres3dnet_brain_parcellation/
                settings_inference.txt
                Modality0.csv
                databrain_std_hist_models_otsu.txt
                inference_niftynet_log
                parcellation_output/
                    window_seg_OAS1_0145_MR2_mpr_n4_anon_sbj_111__niftynet_out.nii.gz
                    inferred.csv
                logs/
                models/
                    model.ckpt-33000.meta
                    model.ckpt-33000.data-00000-of-00001
                    model.ckpt-33000.index
        data/
            OASIS/
                OAS1_0145_MR2_mpr_n4_anon_sbj_111.nii.gz
                license

There are three directories under `~/niftynet`:
1. `extensions` is a Python package and contains the [configuration file](https://niftynet.readthedocs.io/en/dev/config_spec.html)
2. `models` contains the landmarks for [histogram standardization](https://ieeexplore.ieee.org/document/836373) and the network parameters
3. `data` contains an [OASIS](https://www.oasis-brains.org/) MRI that can be used to test the model

```python
models_dir = niftynet_dir / 'models'
zoo_entry = 'highres3dnet_brain_parcellation'
checkpoint_name = 'model.ckpt-33000'
checkpoint_path = models_dir / zoo_entry / 'models' / checkpoint_name
data_dir = niftynet_dir / 'data' / 'OASIS'
config_path = niftynet_dir / 'extensions' / zoo_entry / 'highres3dnet_config_eval.ini'
histogram_landmarks_path = models_dir / zoo_entry / 'databrain_std_hist_models_otsu.txt'
tempdir = Path(tempfile.gettempdir()) / 'miccai_niftynet_pytorch'
tempdir.mkdir(exist_ok=True)
csv_tf_path = tempdir / 'variables_tf.csv'
state_dict_tf_path = tempdir / 'state_dict_tf.pth'
state_dict_pt_path = tempdir / 'state_dict_pt.pth'
prediction_pt_dir = tempdir / 'prediction'
prediction_pt_dir.mkdir(exist_ok=True)

pd.set_option('display.max_colwidth', -1)  # do not truncate strings when displaying data frames
pd.set_option('display.max_rows', None)    # show all rows
```

Note that the path to the checkpoint is not a real filepath but the basename of the three checkpoint files.

## Transfer the parameters

### TensorFlow world

Let's see what variables are stored in the checkpoint.

Some of them are filtered out by `tf2pt.checkpoint_to_state_dict()` for clarity:
* Variables used by the Adam optimizer during training
* Variables with no shape. They won't help much
* Variables containing `biased` or `ExponentialMovingAverage`. Results using these variables have turned out to be different to the ones produced by NiftyNet

We'll store the variables names in a data frame to list them in this notebook and the values in a Python dictionary to retrieve them later.

I figured out the code in `tf2pt.checkpoint_to_state_dict` reading the corresponding [TensorFlow docs](https://www.tensorflow.org/api_docs/python/tf/train/list_variables) and [Stack Overflow answers](https://stackoverflow.com/search?q=restore+tensorflow).

```python
tf2pt.checkpoint_to_state_dict(checkpoint_path, csv_tf_path, state_dict_tf_path)
data_frame_tf = pd.read_csv(csv_tf_path)
state_dict_tf = torch.load(state_dict_tf_path)
data_frame_tf
```

The layers names and parameters shapes overall seem to be coherent with the figure in the paper, but there's an additional $1 \times 1 \times 1$ convolutional layer with 80 output channels. It's also in the [code](https://github.com/NifTK/NiftyNet/blob/1832a516c909b67d0d9618acbd04a7642c12efca/niftynet/network/highres3dnet.py#L93). It seems to be the model with [dropout](http://jmlr.org/papers/v15/srivastava14a.html) used in the study to compute the model's uncertainty, so [our implementation of the architecture](https://github.com/fepegar/highresnet/blob/f434266a51924681f95b01a0f03611fbf1148db6/highresnet/highresnet.py#L82-L97) should include this layer as well.

There are three [dilation](https://arxiv.org/abs/1511.07122) blocks composed of three [residual](https://arxiv.org/abs/1512.03385) blocks each, which have two convolutional layers inside. That's $3 \times 3 \times 2 = 18$ layers. The other three layers are the initial convolution before the first residual block, the conv layer before the dropout and the classifier at the end. Apparently, *all* the convolutional layers have an associated [batch normalization]((https://arxiv.org/abs/1502.03167)) layer. That makes 21 convolutional layers and 21 batch normalization layers, all with parameters that must be transferred.

The shape of the weights of each conovolutional kernel represents the three spatial dimensions, the input channels and the output channels: $(D, H, W, C_{in}, C_{out})$

Each batch norm layer contains 4 parameter groups. Moving mean and variance of the batches, and the affine transformation parameters $\gamma$ (scale or *weight*) and $\beta$ (shift or *bias*):

\begin{align}
y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
\end{align}

Therefore the total number of parameter groups is $21 + 21 \times 4 = 105$. The convolutional layers don't use a bias parameter, as it's not necessary when using batch norm.

### PyTorch world

```python
num_input_modalities = 1
num_classes = 160
model = HighRes3DNet(num_input_modalities, num_classes, add_dropout_layer=True)
```

Let's see what are the variable names created by PyTorch:

```python
state_dict_pt = model.state_dict()
rows = []
for name, parameters in state_dict_pt.items():
    if 'num_batches_tracked' in name:  # filter out for clarity
        continue
    shape = ', '.join(str(n) for n in parameters.shape)
    row = {'name': name, 'shape': shape}
    rows.append(row)
df_pt = pd.DataFrame.from_dict(rows)
df_pt
```

We can see that in Pytorch, `moving_mean` and `moving_variance` are `running_mean` and `running_var`. Also, $\gamma$ and $\beta$ are called `weight` and `bias`.

The convolutional kernels have a different arrangement: $(C_{out}, C_{in}, D, H, W)$.

The names and shapes look coherent and there are 105 lines in both lists, so we should be able to create a mapping between the TensorFlow and the PyTorch variables. The function `tf2pt` receives a TensorFlow-like variable and returns the corresponding PyTorch-like variable.

```python
for name_tf, tensor_tf in state_dict_tf.items():
    shape_tf = tuple(tensor_tf.shape)
    print(f'{str(shape_tf):18}', name_tf)

    # Convert TensorFlow name to PyTorch name
    name_pt, tensor_pt = tf2pt.tf2pt(name_tf, tensor_tf)

    shape_pt = tuple(state_dict_pt[name_pt].shape)
    print(f'{str(shape_pt):18}', name_pt)

    # Sanity check
    if sum(shape_tf) != sum(shape_pt):
        raise ValueError

    state_dict_pt[name_pt] = tensor_pt
    print()
torch.save(state_dict_pt, state_dict_pt_path)
print('State dictionary saved to', state_dict_pt_path)
```

If PyTorch is happy when loading our state dict into the model, we should be on the right track 🤞...

```python
model.load_state_dict(state_dict_pt)
```

No incompatible keys. Yay! 🎉

### Plot weights

Something great about PyTorch is that the modules parameters are easily accesible. Let's plot some of them before and after training:

```python
model_pretrained = model
model_init = HighRes3DNet(num_input_modalities, num_classes, add_dropout_layer=True)
```

By [default](https://github.com/pytorch/pytorch/blob/77353636de32a207cf0a332395f91011bc2f07fb/torch/nn/modules/conv.py#L48-L53), convolutional layers in PyTorch are initialized using [He uniform variance scaling](https://arxiv.org/abs/1502.01852):

```python
visualization.plot_all_parameters(model_init)
```

And this is what they look like after training:

```python
visualization.plot_all_parameters(model_pretrained)
```

## Run inference

### Configuration file

We need to match the configuration used during training in order to obtain consistent results. This are the relevant contents of the downloaded [configuration file](https://niftynet.readthedocs.io/en/dev/config_spec.html):

```ini
[Modality0]
path_to_search = data/OASIS/
filename_contains = nii
pixdim = (1.0, 1.0, 1.0)
axcodes = (R, A, S)

[NETWORK]
name = highres3dnet
volume_padding_size = 10
whitening = True
normalisation = True
normalise_foreground_only=True
foreground_type = mean_plus
histogram_ref_file = databrain_std_hist_models_otsu.txt
cutoff = (0.001, 0.999)

[INFERENCE]
border = 2
spatial_window_size = (128, 128, 128)
```

```python
config = ConfigParser()
config.read(config_path);
```

### Reader

We can figure out the necessary preprocessing reading the paper, the [code](https://github.com/NifTK/NiftyNet/blob/61f2a8bbac1348591412c00f55d1c19b91c0367f/niftynet/application/segmentation_application.py#L95-L192) and the [configuration file](https://niftynet.readthedocs.io/en/dev/config_spec.html) that was downloaded. NiftyNet offers some powerful I/O tools. We will use readers, samplers and aggregators to read and write the necessary files. Some of the [demos](https://github.com/NifTK/NiftyNet/tree/dev/demos/module_examples) show how they can be used.

```python
input_dict = dict(
    path_to_search=str(data_dir),
    filename_contains='nii',
    axcodes=('R', 'A', 'S'),
    pixdim=(1, 1, 1),
)
data_parameters = {
    'image': input_dict,
}
reader = ImageReader().initialise(data_parameters)
```

```python
_, image_data_dict, _ = reader()
original_image = image_data_dict['image']
original_image.shape
```

    (160, 256, 256, 1, 1)

Looking at the shape of our image and knowing it's now in [RAS+ orientation](http://www.grahamwideman.com/gw/brain/orientation/orientterms.htm), we can see that it represents $160$ sagittal slices of $256 \times 256$ pixels, with $1$ channel (monomodal) and $1$ time point. Let's see what it looks like:

```python
from importlib import reload
reload(visualization)
```

    <module 'visualization' from '/home/fernando/git/tf2pt/visualization.py'>

```python
visualization.plot_volume(original_image, title='Original volume')
```

```python
visualization.plot_histogram(original_image, kde=False, labels=True, ylim=(0, 1e6))
```

### Preprocessing

We [pad the input volume](https://niftynet.readthedocs.io/en/dev/window_sizes.html#volume-padding-size) and crop the output volume to reduce the border effect introduced by the padded convolutions:

```python
volume_padding_layer = PadLayer(
    image_name=['image'],  # https://github.com/NifTK/NiftyNet/blob/61f2a8bbac1348591412c00f55d1c19b91c0367f/niftynet/layer/pad.py#L52
    border=(10, 10, 10),
)
```

We use a masking function in order to use only the foreground voxels for normalization:

```python
binary_masking_func = BinaryMaskingLayer(type_str=config['NETWORK']['foreground_type'])
mask = binary_masking_func(original_image)
visualization.plot_volume(mask, enhance=False, title='Binary mask for preprocessing')
```

We use [MRI histogram standardization](https://ieeexplore.ieee.org/document/836373) trained on the training dataset for our test image. We use the mean intensity of the volume as a threshold for the mask, as the authors claim that this usually gives good results.

```python
hist_norm = HistogramNormalisationLayer(
    image_name='image',
    modalities=['Modality0'],
    model_filename=str(histogram_landmarks_path),
    binary_masking_func=binary_masking_func,
    cutoff=(0.001, 0.999),
    name='hist_norm_layer',
)
```

Finally, we set our image foreground to have zero mean and unit variance:

```python
whitening = MeanVarNormalisationLayer(
    image_name='image', binary_masking_func=binary_masking_func)
preprocessing_layers = [
    volume_padding_layer,
    hist_norm,
    whitening,
]
```

Here's our preprocessed image:

```python
reader = ImageReader().initialise(data_parameters)
reader.add_preprocessing_layers(preprocessing_layers)
_, image_data_dict, _ = reader()
preprocessed_image = image_data_dict['image']
visualization.plot_volume(preprocessed_image, title='Preprocessed image')
```

We can clearly see the effect of the whitening layer on the histogram:

```python
visualization.plot_histogram(preprocessed_image, kde=False, ylim=(0, 1e6))
```

### Sampler and aggregator

As the whole image doesn't fit in most GPUs, we need to use a [patch-based](https://niftynet.readthedocs.io/en/dev/window_sizes.html) approach.

We'll use NiftyNet's grid sampler to get all windows from the volume (blue in the previous image) and a grid samples aggregator (red) to reconstruct the output image from the inferred windows. If you have any memory issues, try reducing the window size.

The [window border](https://niftynet.readthedocs.io/en/dev/window_sizes.html#border) is needed to reduce the border effect in a dense prediction.

```python
window_size = 128
window_size = 3 * (window_size, )
window_border = 2, 2, 2
window_size_dict = {'image': window_size}
batch_size = 1

sampler = GridSampler(
    reader,
    window_size_dict,
    window_border=window_border,
)

aggregator = GridSamplesAggregator(
    image_reader=reader,
    window_border=window_border,
    output_path=prediction_pt_dir,
)
```

Now, the most important part: run the parcellation! We'll iterate over the windows provided by the sampler, pass them through the network and aggregate them in the output volume (this might take a couple of minutes):

```python
device = torch.device('cuda') if torch.cuda.is_available() else 'cpu'
model.to(device)
model.eval()
for batch_dict in sampler():
    input_tensor = tf2pt.niftynet_batch_to_torch_tensor(batch_dict).to(device)
    with torch.no_grad():
        logits = model(input_tensor)
    labels = tf2pt.torch_logits_to_niftynet_labels(logits)
    window_dict = dict(window=labels)
    aggregator.decode_batch(window_dict, batch_dict['image_location'])

# Release GPU memory
del model
del model_pretrained
del input_tensor
del logits
torch.cuda.empty_cache()
```

Let's [run the inference using NiftyNet](https://github.com/NifTK/NiftyNetModelZoo/tree/5-reorganising-with-lfs/highres3dnet_brain_parcellation#generating-segmentations-for-example-data) as well, so that we can compare both results:

```python
# %%capture
!net_segment inference -c ~/niftynet/extensions/highres3dnet_brain_parcellation/highres3dnet_config_eval.ini
```

## Check results

```python
input_image = utils.get_first_array(data_dir)
labels_nn = utils.get_first_array(models_dir).astype(np.uint16)
labels_pt = utils.get_first_array(prediction_pt_dir).astype(np.uint16)
```

### Quantitatively

```python
difference = labels_nn != labels_pt
np.count_nonzero(difference)
```

    0

Success! ✨ Both parcellations are exactly the same.

### Qualitatively

```python
colors_path = 'GIFNiftyNet.ctbl'
visualization.plot_volume(
    labels_nn,
    enhance=False,
    colors_path=colors_path,
    title='Parcellation inferred by NiftyNet',
)
visualization.plot_volume(
    labels_pt,
    enhance=False,
    colors_path=colors_path,
    title='Parcellation inferred by PyTorch',
)
```

## Conclusion

## Reproducibility

### Running this notebook

To write this notebook I have used Ubuntu 18.04 installed on an Alienware 13 R3 laptop, which includes a 6 GB GeForce GTX 1060 NVIDIA GPU. I'm using CUDA 9.0.

Inference using PyTorch took 5725 MB of GPU memory. TensorFlow usually takes as much as possible beforehand.

To run this notebook locally, I recommend downloading the repository and creating a [`conda`](https://docs.conda.io/en/latest/miniconda.html) environment:

```shell
$ git clone https://github.com/fepegar/miccai-educational-challenge-2019.git
$ cd miccai-educational-challenge-2019
$ conda create -n emiccai python=3.6 -y
$ conda activate emiccai && conda install jupyterlab -y && jupyter lab
```

```python

```

* Zenodo
* conda
* figshare
* google colab
* github
* Docker


Physics Research and Education Bootcamp - Computational | Episode 1

License: CC BY-NC-SA  https://creativecommons.org/licenses/by-nc-sa/4.0/

Last Modification: Sept. 3, 2021

# ***Preambulo***

Primero carguemos los paquetes de Python necesarios para trabajar.

```python
import numpy as np
import matplotlib.pyplot as plt
from tqdm import tqdm
```

# **Transporte de Neutrones con *Montecarlo***

Hoy aprenderemos como usar el método Montecarlo para estudiar propiedades estadísticas de sistemas con muchas partículas. El procedimiento usado aquí puede ser empleado en otras áreas. Lo importante es tener en mente la clase de conclusiones que pueden obtenerse a través de Montecarlo y su manera de implementación.

# **Teoría**

El problema del transporte de neutrones dentro de un medio es sumamente complejo. Formalmente se puede describir la evolución de un sistema de muchas particulas a traves de la ecuación de Boltzmann, una ecuación diferencial que toma en cuenta el medio en el cual los neutrones se encuentran, su posición, velocidad y momento angular.

\\

Una manera de solucionar este problema es a traves de la aproximación de difusión. En vez de considerar cada párticula de manera independiente, en la ecuación de difusión se aproxima el conjunto de neutrones como un "gas de neutrones". De esta manera se puede describir el comportamiento global del ensamble en términos de propiedades estadísticas como el ***camino libre medio*** o ***longitud de difusión***. Esta ecuación se conoce como la ***Ecuación de Difusión***:
\begin{align}
D\Delta\phi - \Sigma_a\phi = \dfrac{1}{v}\dfrac{d\phi}{dt}
\end{align}
\\

En esta descripción, hablamos de la ***Densidad de Flujo de Neutrones*** ($\phi$), la velocidad media ($v$) , el coeficiente de difusión ($D$) y la sección eficaz macroscópica de absorción ($\Sigma_a$).

\\
Esta ecuación tiene varias soluciones, dependiendo de la geometría y el medio de difusiónn. Hoy nos enfocaremos en medios multiplicadores, es decir, medios en los cuales el número de neutrones aumenta con el tiempo. En este caso, es necesario agregar un término a la ecuaciún para tomar en cuenta la producción de neutrones.

Este mecanismo de producción es la fisión nuclear, y denotamos por $\sigma_f$ la probabilidad de que un neutrón produzca una fisión por unidad de area. Si tomamos en cuenta la densidad del medio, podemos transformar esta cantidad en la sección eficaz macroscópica de fisión ($\Sigma_f$). Si ademas, por cada fisión se producen en promedio ($\nu$) nuevos neutrones:
\begin{align}
D\Delta\phi - \Sigma_a\phi + \nu\Sigma_f\phi = \dfrac{1}{v}\dfrac{d\phi}{dt}
\end{align}
En el caso de criticalidad, el número de neutrones en el sistema se mantiene constante y tenemos:
\begin{align}
D\Delta\phi - \Sigma_a\phi + \nu\Sigma_f\phi = 0
\end{align}
En el caso de un sistema esférico podemos determinar el radio del sistema para que la condición de criticalidad se cumpla:
\begin{align}
R = \dfrac{\pi}{Bg}
\end{align}
con:
\begin{align}
Bg^2 = \dfrac{\nu\Sigma_f - \Sigma_a}{D}
\end{align}

En el caso de que la densidad de neutrones no se mantenga constante (aumente o disminuya) la solucion lamentablemente no es tan sencilla de computar.

Podemos sin embargo, hablar de la cantidad $K_\mathrm{eff}$ que nos permite cuantificar la diferencia en el número de neutrones entre una generación y otra.

\begin{align}
K_\mathrm{eff} = \frac{\mathrm{numero\,\, de\,\, neutrones \,\,actual}}{\mathrm{numero\,\,de\,\,neutrones\,\,previo}}
\end{align}

De esta manera si:

* $K_\mathrm{eff}$ > 1 : hablamos de un sistema supercrítico, la densidad de neutrones aumenta exponencialmente
* $K_\mathrm{eff}$ = 1 : hablamos de un sistema crítico, la reacción es autosostenible y la cantidad de neutrones permance constante
* $K_\mathrm{eff}$ < 1 : hablamos de un sistema subcrítico, la densidad de neutrones disminuye con el tiempo

\\

Verificar si un sistema dado es crítico o supercrítico no es trivial. Para esto podemos utilizar el metodo Montecarlo y asi estimar esta proporción de neutrones de una generación a otra.

\\

Nota: para el caso que vamos a estudiar:
*  $\Sigma_a$ es la sección eficaz macroscópica de absorción y se refiere a la probabilidad total de que el neutrón sea absorbido y produzca fisión o simplemente active el núcleo que lo recibe.
* $\Sigma_f$ es la seccion eficaz macroscópica de fisión y se refiere a la probabilidad de que el neutrón absorbido produzca fisión
* $\Sigma_t$ es la probabilidad total de interacción, en el contexto de Difusión podemos entenderlo como el camino libre medio de un neutrón
* $\Sigma_s$ es la probabilidad de que el neutrón que interacciona sea desviado

\\

Es conveniente tener en mente que:

\begin{align}
\Sigma_t = \Sigma_s + \Sigma_a
\end{align}

Las unidades de los $\Sigma_j$ son de $cm^{-1}$

# Práctica

Necesitamos un código que nos genere inicialmente neutrones, los mueva a traves del medio y determine si despues de un movimiento ocurre una fisión, absorción o scattering.

```python
def generar_neutrones(N_neutrones,radio):

    #Inicializamos las posiciones iniciales de los neutrones dentro de la esfera
    # Generamos dos coordenadas (r,theta) y luego a traves de las propiedades trigonometricas
    # luego convertimos estas coordenadas polares a cartesianas.
    r0 = np.random.uniform(0,radio,size=N_neutrones)
    theta0 = np.random.uniform(0,2*np.pi-.01,size=N_neutrones)
    x0 = r0*np.cos(theta0)
    y0 = r0*np.sin(theta0)
    # Guardamos las posiciones iniciales dentro de un arreglo para uso posterior
    fission_sites = np.array([x0,y0]).T
    return fission_sites
r = 2
Posiciones = generar_neutrones(1000,r)

plt.figure()
plt.suptitle('Distribucion de puntos iniciales de neutrones',fontweight='bold')
plt.scatter(Posiciones[:,0],Posiciones[:,1])
plt.xlabel('x',fontweight='bold')
plt.ylabel('y',fontweight='bold')
plt.xlim(-r-1,r+1)
plt.ylim(-r-1,r+1)
plt.grid()
```

Durante el transporte de un neutrón, este puede ser absorbido, desviado o ocasionar una fisión. Si el sistema es supercrítico el programa colapsará debido a límites de memoria ya que en cada ciclo de transporte habra más y más neutrones que seguir. Para evitar esto asignaremos pesos a los neutrones, con los cuales simbolizaremos el aumento o disminución de la cantidad de neutrones.

\\

Al final de cada ciclo, contamos el número de fisiones producidas y la posición de interacción. En el siguiente ciclo iniciamos nuevos neutrones desde estos punto de interacción y seguimos su evolución. Tomando el cociente entre los pesos de los neutrones de un ciclo entre el ciclo previo podemos obtener $K_\mathrm{eff}$

\\

Necesitamos ahora un código que mueva un neutrón aleatoreamente por el espacio:

Para el transporte debemos considerar $\Sigma_a$, $\Sigma_t$, $\Sigma_f$ y $\nu$

```python
def random_transport(x,y,Sigma_t,niter):
  pos = [[x,y]]
  for i in range(niter):
    new_r = -np.log(1-np.random.random())/Sigma_t # Ya que tenemos solo una distancia promedio necesitamos muestrear las distancias
    new_theta = np.random.uniform(0,2*np.pi-.01) # Muestramos aleatoriamente un angulo
    x += new_r*np.cos(new_theta)
    y += new_r*np.sin(new_theta)
    pos.append([x,y])
  return pos

Posiciones = random_transport(np.random.random(),np.random.random(),1,100)
Posiciones = np.array(Posiciones)
plt.figure()
plt.suptitle('Transporte aleatorio de un neutron',fontweight='bold')
plt.plot(Posiciones[:,0],Posiciones[:,1])
plt.scatter(Posiciones[0,0],Posiciones[0,1],label='principio',s=40,c='g')
plt.scatter(Posiciones[-1,0],Posiciones[-1,1],label='final',s=40,c='r')
plt.xlabel('x',fontweight='bold')
plt.ylabel('y',fontweight='bold')
plt.grid()
plt.legend()

```

Ahora necesitamos una lógica para determinar si el neutrón interactúa o no.
Para esto hacemos uso de las secciones eficaces macroscópicas.

\\
Como hemos visto, para generar muestras de una distribución debemos invertir la ***funcion de densidad cumulativa***. En casos como el nuestro en los cuales esta función no es conocida, tenemos que usar otros métodos para generar muestras.

El mecanismo que usaremos es el ***rejection sampling***. En terminos sencillos consiste en generar un número aleatorio y compararlo a la probabilidad de que ocurra algun evento. Si la probabilidad es menor, el evento ocurre, si es mayor el evento no ocurre y se procede a evaluar si otro evento distinto ocurrió mediante el mismo mecanismo.

\\
\begin{align}
\alpha \in U[0,1] \rightarrow p(\alpha) \begin{cases}
1    &\text{si } \alpha \leq \dfrac{p_\mathrm{evento}}{p_\mathrm{total}} \\
0    &\text{si } \alpha > \dfrac{p_\mathrm{evento}}{p_\mathrm{total}}\\
\end{cases}
\end{align}

\\

Ahora escribimos un código que tome esto en cuenta, lo usaremos para estudiar si durante el trayecto el neutrón colisiona y es desviado, es absorbido o produce fisión.

\\

```python
def random_transport(x,y,Sigma_t,Sigma_a,Sigma_f,niter):
  # Calculamos la seccion eficaz macroscopica de desvio
  Sigma_s = Sigma_t - Sigma_a
  Pos = [[x,y]]
  alive = 1 # Usamos esta variable como un flag para determinar si debemos continuar siguiendo la evolucion de la particula
  Fision = 0
  while alive == 1:
    for i in range(niter):
      new_r = -np.log(1-np.random.random())/Sigma_t # Ya que tenemos solo una distancia promedio necesitamos muestrear las distancias
      new_theta = np.random.uniform(0,2*np.pi-.01) # Muestramos aleatoriamente un angulo
      x += new_r*np.cos(new_theta)
      y += new_r*np.sin(new_theta)
      Pos.append([x,y])

      # Verificamos si ocurre una interaccion:
      coll_prob= np.random.random()
      if coll_prob <= Sigma_s/Sigma_t: # La fraccion de la probabidad total de interaccion que ocurra un desvio
        continue # No debemos cambiar nada, proseguimos con la siguiente iteracion.
      else:
        # No hubo desvio (scattering), estudiamos ahora si hubo fision o simple absorcion
        fiss_prob = np.random.random()
        alive = 0
        if fiss_prob <= Sigma_f/Sigma_a:
          Fision = 1
          print(f'Ha ocurrido una fision tras {i+1} iteraciones')
          return Pos
        else:
          print(f'El neutron ha sido absorbido tras {i+1} iteraciones')
          return Pos
  return Pos
```

```python
Posiciones = random_transport(np.random.random(),np.random.random(),1,0.3,0.1,100)
Posiciones = np.array(Posiciones)
plt.figure()
plt.suptitle('Transporte aleatorio de un neutron',fontweight='bold')
plt.plot(Posiciones[:,0],Posiciones[:,1])
plt.scatter(Posiciones[0,0],Posiciones[0,1],label='principio',s=40,c='g')
plt.scatter(Posiciones[-1,0],Posiciones[-1,1],label='final',s=40,c='r')
plt.xlabel('x',fontweight='bold')
plt.ylabel('y',fontweight='bold')
plt.grid()
plt.legend()

```

Solo nos falta crear un loop grande que genere ciclos de transporte con los neutrones

\\

El algoritmo que queremos debe:

1.   Generar un grupo de N neutrones distribuidos al azar en el espacio
2.   Seguir la evolución de cada neutron
3.   Al final del ciclo tomar nota de los lugares del espacio donde hubo una interacción y ajustar el vector de pesos de manera proporcional al número de fisiones occuridas.
4.   Comenzar el siguiente ciclo g con N neutrones distribuidos entre los puntos de interacción del ciclo previo.

 Nota:

Como todo algoritmo numérico, es probable que el valor estimado de $K_\mathrm{eff}$ varíe de un ciclo a otro. Ademas, ya que no tenemos la CDF subyacente de las interacciones y la aproximamos a traves de ***rejection sampling***, es necesario realizar varias iteraciones hasta que la distribución de probabilidad de interacción se vuelva estacionaria y converja asintóticamente a la distribución real. Para esto consideramos un número de ciclos extras, los cuales no consideraremos válidos para estimar $K_\mathrm{eff}$

(Si has visto Cadenas de Markov, esta noción te parecera familiar a la del ***burn in*** de la traza)

### Estimando K_eff

Reordenando y combinando lo que hemos escrito:

```python
def homog_sphere_k(N,Sig_t,Sig_f,Sig_a,nu,radio,inactive_cycles = 5, active_cycles = 20):

    Sig_s = Sig_t-Sig_a # Calculamos la seccion eficaz macroscopica de desvio

    #Inicializamos las posiciones iniciales de los neutrones dentro de la esfera
    # Generamos dos coordenadas (r,theta) y luego a traves de las propiedades trigonometricas
    # luego convertimos estas coordenadas polares a cartesianas.
    r0 = np.random.uniform(0,radio,size=N)
    theta0 = np.random.uniform(0,2*np.pi-.01,size=N)
    x0 = r0*np.cos(theta0)
    y0 = r0*np.sin(theta0)
    # Guardamos las posiciones iniciales dentro de un arreglo para uso posterior
    fission_sites = np.array([x0,y0]).T
    posiciones = fission_sites.copy()
    # Inicializamos un vector de pesos asignando el numero medio de neutrones por fision
    weights = nu*np.ones(N)
    old_gen = np.sum(weights)
    k = np.zeros(inactive_cycles+active_cycles)

    Fs = {}

    for cycle in tqdm(range(inactive_cycles+active_cycles)):
        fission_sites = np.empty((1,2)) # Creamos un array vacio 1x2
        fission_site_weights = np.empty(1) # Creamos un array vacio 1x1
        for neut in range(weights.size):
            #Tomamos un neutron de la lista de neutrones
            posicion = posiciones[neut] # posicion del neutron
            weight = weights[neut] # peso asociado
            alive = 1 # Indicamos que esta 'vivo' y hay que seguir su evolucion
            while (alive):
                # Muestramos la distancia a la proxima colision
                new_r = -np.log(1-np.random.random())/Sig_t # Ya que tenemos solo una distancia promedio necesitamos muestrear las distancias
                new_theta = np.random.uniform(0,2*np.pi-.01) # Muestramos aleatoriamente un angulo
                # Actualizamos la posicion del neutron
                posicion[0] += new_r*np.cos(new_theta)
                posicion[1] += new_r*np.sin(new_theta)
                # Verificamos si aun estamos dentro de la esfera
                if (posicion[0]**2 + posicion[1]**2) > radio**2:
                    alive = 0 # El neutron ha salido de la esfera, dejamos de seguirlo
                else:
                    # Tomamos una muestra de la probabilidad de interaccion
                    coll_prob = np.random.random()
                    if (coll_prob < Sig_s/Sig_t):
                      # Si la probabilidad es menor que la probabilidad de desvio el neutron no interactuo,
                      # recalculamos la distancia hasta la siguiente colision
                        continue
                    else:
                      # Si el neutron colisiono, pudo haber ocasinado una fision o ser absorvido
                      # para esto tomamos una muestra de la probabilidad de fision
                        fiss_prob = np.random.random()
                        alive = 0 #
                        if (fiss_prob <= Sig_f/Sig_a):
                          # Si la probabilidad de fision es menor o igual al cociente
                          # entre las secciones eficaces macroscopicas, consideramos que ocurre una fission
                          # en la posicion actual

                            # agregamos la posicion actual al arreglo de los lugares de fision
                            fission_sites = np.vstack((fission_sites,posicion.reshape(-1,1).T))
                            # agregamos el peso de la particula correspondiente al arreglo de pesos
                            fission_site_weights = np.append(fission_site_weights,weight)
                          # En caso de no haber fision, el neutron es absorbido. En nuestra approximacion
                          # no hacemos nada, sin embargo para mayor fidelidad, deberiamos actualizar la composicion
                          # atomica del medio y ajustar las correspondientes secciones eficaces.

        # Finalizamos un ciclo

        # Borramos la primera entrada del arreglo ya que contiene datos dummy
        fission_sites = np.delete(fission_sites,0,axis=0)
        # Realizamos el mismo procedimiento para este otro arreglo
        fission_site_weights = np.delete(fission_site_weights,0,axis=0)
        Fs[str(cycle)] = fission_sites
        # Determinamos el numero de neutrones a inicializar en el proximo ciclo
        # desde los puntos de interaccion determinados en el ciclo anterior
        num_per_site = int(np.ceil(N/fission_sites.shape[0]))
        # Reiniciamos el vector de posiciones de las particulas:
        posiciones = np.empty((1,2))
        weights = np.empty(1)
        # Creamos los nuevos arreglos de posiciones y pesos para usarlos en el proximo ciclo
        for site in range(fission_sites.shape[0]):
            site_pos = fission_sites[site]
            site_weight = fission_site_weights[site]
            posiciones = np.vstack((posiciones,
            site_pos*np.ones((num_per_site,1))))
            weights = np.append(weights,
            site_weight * nu/num_per_site*np.ones((num_per_site,1)))
        # Borramos la primera entrada de los arreglos al igual que hicimos arriba
        posiciones = np.delete(posiciones,0,axis=0)
        weights = np.delete(weights,0,axis=0)
        # Estimamos el numero de neutrones producidos en la generacion actual
        new_gen = np.sum(weights)
        # Calculamos el valor de Keff correspondiente para el ciclo actual
        k[cycle] = new_gen/old_gen
        old_gen = new_gen
    return k, Fs
```

Ahora aplicamos nuestro cídigo a una esfera de radio 3cm usando los siguientes parámetros:

* $\Sigma_t = $ `0.3382197087866109`
* $\Sigma_f = $ `0.0920156560669456`
* $\Sigma_a = $ `0.1049475861087866`
* $\nu = $ `2.98`
* $N$ = `50e3` neutrones por ciclo
* Ciclos Activos = `100`
* Ciclos Inactivos = `10`

```python
St = 0.3382197087866109
Sf = 0.0920156560669456
Sa = 0.1049475861087866
nu = 2.98
r = 3
Nciclos_act = 100
Nciclos_inact = 10
N_neutrones = 50000

K, Fs = homog_sphere_k(N_neutrones,St,Sf,Sa,nu,r,Nciclos_inact,Nciclos_act)
Kmean = np.mean(K[Nciclos_inact:])
Kstd = np.std(K[Nciclos_inact:])
print('Keff = {:.4f} +/- {:.4f}'.format(Kmean,Kstd))
```

    100%|██████████| 110/110 [05:42<00:00,  3.12s/it]

    Keff = 0.6910 +/- 0.0058

```python
plt.figure()
plt.suptitle(r'Evolucion de $K_{eff}$ vs Ciclo de Transporte',fontweight='bold')
plt.plot(np.arange(Nciclos_act+Nciclos_inact),K)
plt.ylabel(r'K_${eff}$',fontweight='bold')
plt.xlabel('Ciclo',fontweight='bold')
plt.grid()

```

# Tarea

* Determine el radio crítico de una esfera de $^{239}\mathrm{Pu}$. Cual es la masa crítica?

* Modifique el cídigo para considerar ahora una esfera hueca con radio interior $R_{in}$ y radio exterior $R_{ext}$. Determine valores de estos radios para que el sistema sea crítico a densidad normal.

Version 1.0, Autor : Antonio Figueroa

```python
#libs summary
import types
def imports():
    for name, val in globals().items():
        if isinstance(val, types.ModuleType):
            yield val.__name__
list(imports())
```

    ['builtins',
     'builtins',
     'IPython.core.shadowns',
     'numpy',
     'matplotlib.pyplot',
     'types']


# Text Classification with SpaCy

A common task in NLP is **text classification**. This is "classification" in the conventional machine learning sense, and it is applied to text. Examples include spam detection, sentiment analysis, and tagging customer queries.

In this tutorial, you'll learn text classification with spaCy. The classifier will detect spam messages, a common functionality in most email clients. Here is an overview of the data you'll use:

```python
import pandas as pd

# Loading the spam data
# ham is the label for non-spam messages
spam = pd.read_csv('../input/nlp-course/spam.csv')
spam.head(10)
```

      label
      text

      0
      ham
      Go until jurong point, crazy.. Available only ...

      1
      ham
      Ok lar... Joking wif u oni...

      2
      spam
      Free entry in 2 a wkly comp to win FA Cup fina...

      3
      ham
      U dun say so early hor... U c already then say...

      4
      ham
      Nah I don't think he goes to usf, he lives aro...

      5
      spam
      FreeMsg Hey there darling it's been 3 week's n...

      6
      ham
      Even my brother is not like to speak with me. ...

      7
      ham
      As per your request 'Melle Melle (Oru Minnamin...

      8
      spam
      WINNER!! As a valued network customer you have...

      9
      spam
      Had your mobile 11 months or more? U R entitle...

# Bag of Words
Machine learning models don't learn from raw text data. Instead, you need to convert the text to something numeric.

The simplest common representation is a variation of one-hot encoding. You represent each document as a vector of term frequencies for each term in the vocabulary. The vocabulary is built from all the tokens (terms) in the corpus (the collection of documents).

As an example, take the sentences "Tea is life. Tea is love." and "Tea is healthy, calming, and delicious." as our corpus. The vocabulary then is `{"tea", "is", "life", "love", "healthy", "calming", "and", "delicious"}` (ignoring punctuation).

For each document, count up how many times a term occurs, and place that count in the appropriate element of a vector. The first sentence has "tea" twice and that is the first position in our vocabulary, so we put the number 2 in the first element of the vector. Our sentences as vectors then look like

$$
\begin{align}
v_1 &= \left[\begin{matrix} 2 & 2 & 1 & 1 & 0 & 0 & 0 & 0 \end{matrix}\right] \\
v_2 &= \left[\begin{matrix} 1 & 1 & 0 & 0 & 1 & 1 & 1 & 1 \end{matrix}\right]
\end{align}
$$

This is called the **bag of words** representation. You can see that documents with similar terms will have similar vectors. Vocabularies frequently have tens of thousands of terms, so these vectors can be very large.

Another common representation is **TF-IDF (Term Frequency - Inverse Document Frequency)**. TF-IDF is similar to bag of words except that each term count is scaled by the term's frequency in the corpus. Using TF-IDF can potentially improve your models. You won't need it here. Feel free to look it up though!

# Building a Bag of Words model

Once you have your documents in a bag of words representation, you can use those vectors as input to any machine learning model. spaCy handles the bag of words conversion and building a simple linear model for you with the `TextCategorizer` class.

The TextCategorizer is a spaCy **pipe**. Pipes are classes for processing and transforming tokens. When you create a spaCy model with `nlp = spacy.load('en_core_web_sm')`, there are default pipes that perform part of speech tagging, entity recognition, and other transformations. When you run text through a model `doc = nlp("Some text here")`, the output of the pipes are attached to the tokens in the `doc` object. The lemmas for `token.lemma_` come from one of these pipes.

You can remove or add pipes to models. What we'll do here is create an empty model without any pipes (other than a tokenizer, since all models always have a tokenizer). Then, we'll create a TextCategorizer pipe and add it to the empty model.

```python
import spacy

# Create an empty model
nlp = spacy.blank("en")

# Create the TextCategorizer with exclusive classes and "bow" architecture
textcat = nlp.create_pipe(
              "textcat",
              config={
                "exclusive_classes": True,
                "architecture": "bow"})

# Add the TextCategorizer to the empty model
nlp.add_pipe(textcat)
```

Since the classes are either ham or spam, we set `"exclusive_classes"` to `True`. We've also configured it with the bag of words (`"bow"`) architecture. spaCy provides a convolutional neural network architecture as well, but it's more complex than you need for now.

Next we'll add the labels to the model. Here "ham" are for the real messages, "spam" are spam messages.

```python
# Add labels to text classifier
textcat.add_label("ham")
textcat.add_label("spam")
```

    1

# Training a Text Categorizer Model

Next, you'll convert the labels in the data to the form TextCategorizer requires. For each document, you'll create a dictionary of boolean values for each class.

For example, if a text is "ham", we need a dictionary `{'ham': True, 'spam': False}`. The model is looking for these labels inside another dictionary with the key `'cats'`.

```python
train_texts = spam['text'].values
train_labels = [{'cats': {'ham': label == 'ham',
                          'spam': label == 'spam'}}
                for label in spam['label']]
```

Then we combine the texts and labels into a single list.

```python
train_data = list(zip(train_texts, train_labels))
train_data[:3]
```

    [('Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...',
      {'cats': {'ham': True, 'spam': False}}),
     ('Ok lar... Joking wif u oni...', {'cats': {'ham': True, 'spam': False}}),
     ("Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's",
      {'cats': {'ham': False, 'spam': True}})]

Now you are ready to train the model. First, create an `optimizer` using `nlp.begin_training()`. spaCy uses this optimizer to update the model. In general it's more efficient to train models in small batches. spaCy provides the `minibatch` function that returns a generator yielding minibatches for training. Finally, the minibatches are split into texts and labels, then used with `nlp.update` to update the model's parameters.

```python
from spacy.util import minibatch

spacy.util.fix_random_seed(1)
optimizer = nlp.begin_training()

# Create the batch generator with batch size = 8
batches = minibatch(train_data, size=8)
# Iterate through minibatches
for batch in batches:
    # Each batch is a list of (text, label) but we need to
    # send separate lists for texts and labels to update().
    # This is a quick way to split a list of tuples into lists
    texts, labels = zip(*batch)
    nlp.update(texts, labels, sgd=optimizer)
```

This is just one training loop (or epoch) through the data. The model will typically need multiple epochs. Use another loop for more epochs, and optionally re-shuffle the training data at the begining of each loop.

```python
import random

random.seed(1)
spacy.util.fix_random_seed(1)
optimizer = nlp.begin_training()

losses = {}
for epoch in range(10):
    random.shuffle(train_data)
    # Create the batch generator with batch size = 8
    batches = minibatch(train_data, size=8)
    # Iterate through minibatches
    for batch in batches:
        # Each batch is a list of (text, label) but we need to
        # send separate lists for texts and labels to update().
        # This is a quick way to split a list of tuples into lists
        texts, labels = zip(*batch)
        nlp.update(texts, labels, sgd=optimizer, losses=losses)
    print(losses)
```

    {'textcat': 0.43189741921099767}
    {'textcat': 0.6474976215331196}
    {'textcat': 0.7842154536487618}
    {'textcat': 0.8716683716818165}
    {'textcat': 0.9280939335008995}
    {'textcat': 0.9655779922872296}
    {'textcat': 0.9939651840090362}
    {'textcat': 1.0127976631523663}
    {'textcat': 1.0275637812859075}
    {'textcat': 1.0378531470013608}

# Making Predictions

Now that you have a trained model, you can make predictions with the `predict()` method. The input text needs to be tokenized with `nlp.tokenizer`. Then you pass the tokens to the predict method which returns scores. The scores are the probability the input text belongs to the classes.

```python
texts = ["Are you ready for the tea party????? It's gonna be wild",
         "URGENT Reply to this message for GUARANTEED FREE TEA" ]
docs = [nlp.tokenizer(text) for text in texts]

# Use textcat to get the scores for each doc
textcat = nlp.get_pipe('textcat')
scores, _ = textcat.predict(docs)

print(scores)
```

    [[9.9994397e-01 5.6023764e-05]
     [1.1491306e-02 9.8850864e-01]]

The scores are used to predict a single class or label by choosing the label with the highest probability. You get the index of the highest probability with `scores.argmax`, then use the index to get the label string from `textcat.labels`.

```python
# From the scores, find the label with the highest score/probability
predicted_labels = scores.argmax(axis=1)
print([textcat.labels[label] for label in predicted_labels])
```

    ['ham', 'spam']

Evaluating the model is straightforward once you have the predictions. To measure the accuracy, calculate how many correct predictions are made on some test data, divided by the total number of predictions.

# Your Turn
Try it yourself as you **[predict the sentiment of Yelp reviews](https://www.kaggle.com/kernels/fork/6061027)**.

---

*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161466) to chat with other Learners.*