```python
!pip install qiskit
```

    Collecting qiskit
      Downloading qiskit-0.33.0.tar.gz (13 kB)
    Collecting qiskit-terra==0.19.0
      Downloading qiskit_terra-0.19.0-cp37-cp37m-manylinux2010_x86_64.whl (6.4 MB)
         |████████████████████████████████| 6.4 MB 4.7 MB/s
    Collecting qiskit-aer==0.9.1
      Downloading qiskit_aer-0.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (17.9 MB)
         |████████████████████████████████| 17.9 MB 621 kB/s
    Collecting qiskit-ibmq-provider==0.18.1
      Downloading qiskit_ibmq_provider-0.18.1-py3-none-any.whl (237 kB)
         |████████████████████████████████| 237 kB 61.4 MB/s
    Collecting qiskit-ignis==0.7.0
      Downloading qiskit_ignis-0.7.0-py3-none-any.whl (200 kB)
         |████████████████████████████████| 200 kB 59.1 MB/s
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.9.1->qiskit) (1.4.1)
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.9.1->qiskit) (1.19.5)
    Collecting requests-ntlm>=1.1.0
      Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.1->qiskit) (1.24.3)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.1->qiskit) (2.23.0)
    Collecting websocket-client>=1.0.1
      Downloading websocket_client-1.2.3-py3-none-any.whl (53 kB)
         |████████████████████████████████| 53 kB 2.0 MB/s
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.1->qiskit) (2.8.2)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.10.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)
         |████████████████████████████████| 1.4 MB 34.7 MB/s
    Collecting symengine>=0.8
      Downloading symengine-0.8.1-cp37-cp37m-manylinux2010_x86_64.whl (38.2 MB)
         |████████████████████████████████| 38.2 MB 1.3 MB/s
    Collecting stevedore>=3.0.0
      Downloading stevedore-3.5.0-py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.2 MB/s
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.0->qiskit) (1.7.1)
    Collecting python-constraint>=1.4
      Downloading python-constraint-1.4.0.tar.bz2 (18 kB)
    Collecting scipy>=1.0
      Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
         |████████████████████████████████| 38.1 MB 1.3 MB/s
    Collecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (943 kB)
         |████████████████████████████████| 943 kB 45.1 MB/s
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.0->qiskit) (5.4.8)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.0->qiskit) (0.3.4)
    Collecting ply>=3.10
      Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.1 MB/s
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.1->qiskit) (1.15.0)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.1->qiskit) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.1->qiskit) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.1->qiskit) (2021.10.8)
    Collecting cryptography>=1.3
      Downloading cryptography-36.0.0-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
         |████████████████████████████████| 3.6 MB 19.7 MB/s
    Collecting ntlm-auth>=1.0.2
      Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.1->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.1->qiskit) (2.21)
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.0->qiskit) (4.8.2)
    Collecting pbr!=2.1.0,>=2.0.0
      Downloading pbr-5.8.0-py2.py3-none-any.whl (112 kB)
         |████████████████████████████████| 112 kB 60.6 MB/s
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.0->qiskit) (3.10.0.2)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.0->qiskit) (3.6.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.19.0->qiskit) (1.2.1)
    Building wheels for collected packages: qiskit, python-constraint
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.33.0-py3-none-any.whl size=11763 sha256=f90de4a758faa8aa63936577ad98bb51caac39aab1bd172ede9b614391fd1766
      Stored in directory: /root/.cache/pip/wheels/d4/60/ef/c49ae113df02818c4ade5fb6e2e89bb928625412eda0bb25f5
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24081 sha256=652bafd2fd27167bd2a4e6743482be7f82ceac223e922cd6fd72e9aa45c1437c
      Stored in directory: /root/.cache/pip/wheels/07/27/db/1222c80eb1e431f3d2199c12569cb1cac60f562a451fe30479
    Successfully built qiskit python-constraint
    Installing collected packages: pbr, tweedledum, symengine, stevedore, scipy, retworkx, python-constraint, ply, ntlm-auth, cryptography, websocket-client, requests-ntlm, qiskit-terra, qiskit-ignis, qiskit-ibmq-provider, qiskit-aer, qiskit
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed cryptography-36.0.0 ntlm-auth-1.5.0 pbr-5.8.0 ply-3.11 python-constraint-1.4.0 qiskit-0.33.0 qiskit-aer-0.9.1 qiskit-ibmq-provider-0.18.1 qiskit-ignis-0.7.0 qiskit-terra-0.19.0 requests-ntlm-1.1.0 retworkx-0.10.2 scipy-1.7.3 stevedore-3.5.0 symengine-0.8.1 tweedledum-1.1.1 websocket-client-1.2.3

```python
import qiskit
from qiskit.circuit.random import random_circuit
from qiskit import transpile, assemble
```

```python
import matplotlib.pyplot as plt
%matplotlib inline

import torch
import torch.utils.data
from torch.autograd import Function
from torchvision import datasets, transforms
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F

from itertools import combinations
import numpy as np
import matplotlib.pyplot as plt
from __future__ import print_function

import os
import sys
```

```python
#Custom library
'''
sys.path.append("../LQDNN")
sys.path.append("../LQDNN/model")
sys.path.append("../LQDNN/dynamics")
'''
sys.path.append("./sample_data")
from visualize import *
from ode_solver import *
from network import *
from dataloader import *
from lagrangian import *
```

```python
print("Loading training and test data...")
train_data = np.load('/content/sample_data/train_dataset.npz')
train_inputs = train_data["input"]
train_labels = train_data["labels"]

#train_dataset = DynamicsDataset(train_inputs, train_labels)
X_train = train_inputs[:1000]
y_train = train_labels[:1000]

#train_dataset = DynamicsDataset(train_inputs, train_labels)
train_dataset = DynamicsDataset(X_train, y_train)

train_dataloader = torch.utils.data.DataLoader(train_dataset,
                                                 batch_size=1,
                                                 shuffle=True,
                                                 collate_fn=DynamicsDataset.collate_fn,
                                                 pin_memory=True,
                                                 num_workers=1)
test_data = np.load('/content/sample_data/val_dataset.npz')
test_inputs = train_data["input"]
test_labels = train_data["labels"]
#print(len(test_inputs))
X_test = test_inputs[:1000]
y_test = test_labels[:1000]

test_dataset = DynamicsDataset(X_test, y_test)

test_dataloader = torch.utils.data.DataLoader(test_dataset,
                                                 batch_size=1,
                                                 shuffle=False,
                                                 collate_fn=DynamicsDataset.collate_fn,
                                                 pin_memory=True,
                                                 num_workers=1)
print("Loading training and test data completed")
```

    Loading training and test data...
    Loading training and test data completed

```python
for (X_train, y_train) in train_dataloader:
    print('X_train:', X_train.size(), 'type:', X_train.type())
    print('y_train:', y_train.size(), 'type:', y_train.type())
    print('X_train:', X_train)
    print('y_train:', y_train)
    break

for (X_test, y_test) in test_dataloader:
    print('X_test:', X_test.size(), 'type:', X_test.type())
    print('y_test:', y_test.size(), 'type:', y_test.type())
    print('X_test:', X_test)
    print('y_test:', y_test)
    break
```

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    X_train: torch.Size([1, 4]) type: torch.DoubleTensor
    y_train: torch.Size([1, 4]) type: torch.DoubleTensor
    X_train: tensor([[ 0.4006, -2.1999, -0.1462, -0.0383]], dtype=torch.float64)
    y_train: tensor([[-0.1462, -0.0383, -0.6669,  7.3712]], dtype=torch.float64)

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    X_test: torch.Size([1, 4]) type: torch.DoubleTensor
    y_test: torch.Size([1, 4]) type: torch.DoubleTensor
    X_test: tensor([[ 1.9537,  0.2797, -0.0780, -0.2268]], dtype=torch.float64)
    y_test: tensor([[-0.0780, -0.2268, -9.3138, -3.6618]], dtype=torch.float64)

```python
class QCircuit:
    """
    This class defines filter circuit of Quanvolution layer
    """

    def __init__(self, num_qubit, backend, shots):
        # --- Circuit definition start ---
        self.n_qubits = num_qubit
        self._circuit = qiskit.QuantumCircuit(self.n_qubits,2)

        all_qubits = [i for i in range(self.n_qubits)]
        self._circuit.h(all_qubits)

        self.theta = [qiskit.circuit.Parameter('theta{}'.format(i)) for i in range(self.n_qubits)]

        for i in range(self.n_qubits):
            self._circuit.rx(self.theta[i], i)

        self._circuit.cx(0,2)
        self._circuit.cx(1,3)

        self._circuit.barrier()

        self._circuit.measure(2,0)
        self._circuit.measure(3,1)

        # ---- Circuit definition end ----

        self.backend   = backend
        self.shots     = shots

    def run(self, data):
        ListTarget = []
        for valT in data.detach().numpy():
            for j in range(len(valT)):
                ListTarget.append(valT[j])
        param_dict = dict()

        i= 0
        for theta in ListTarget:
            param_dict[self.theta[i]] = theta#[i]
            i += 1
        param_binds = [param_dict]

        t_qc = transpile(self._circuit,
                         self.backend)
        qobj = assemble(t_qc,
                        shots=self.shots,
                        parameter_binds = param_binds)
        job = self.backend.run(qobj)
        result = job.result().get_counts()

        probabilities = []
        counts = 0
        for key, val in result.items():
            cnt = sum([int(char) for char in key])
            counts += cnt * val

            # Compute probabilities for each state
            probab = counts / (self.shots * self.n_qubits)
            probabilities.append(probab)

        return probabilities
```

```python
#Testing the circuit
backend = qiskit.Aer.get_backend('qasm_simulator')
num_qubit = 4
circ = QCircuit(num_qubit, backend, 100)
data = torch.tensor([[-0.2397, -2.9656, -0.1623, -0.0680]])

print(circ.run(data))
circ._circuit.draw()#output='mpl')
```

    [0.0, 0.0625, 0.1175, 0.2475]

<pre style="word-wrap: normal;white-space: pre;background: #fff0;line-height: 1.1;font-family: "Courier New",Courier,monospace">     ┌───┐┌────────────┐           ░
q_0: ┤ H ├┤ Rx(theta0) ├──■────────░───────
     ├───┤├────────────┤  │        ░
q_1: ┤ H ├┤ Rx(theta1) ├──┼────■───░───────
     ├───┤├────────────┤┌─┴─┐  │   ░ ┌─┐
q_2: ┤ H ├┤ Rx(theta2) ├┤ X ├──┼───░─┤M├───
     ├───┤├────────────┤└───┘┌─┴─┐ ░ └╥┘┌─┐
q_3: ┤ H ├┤ Rx(theta3) ├─────┤ X ├─░──╫─┤M├
     └───┘└────────────┘     └───┘ ░  ║ └╥┘
c: 2/═════════════════════════════════╩══╩═
                                      0  1 </pre>

```python
class HybridFunction(Function):
    """ Hybrid quantum - classical function definition """

    @staticmethod
    def forward(ctx, input, quantum_circuit, shift):
        """ Forward pass computation """
        ##print('hFor:', input)
        #print('hFor1:', input[0])
        #print('hFor2:', input[0].tolist())
        ctx.shift = shift
        ctx.quantum_circuit = quantum_circuit
        #expectation_z = []
        #for i in range(len(input)):
        #    expectation_z.append(ctx.quantum_circuit.run(input[i].tolist()))
        #print('expectation_z',expectation_z)
        expectation_z = ctx.quantum_circuit.run(input[0].tolist())
        result = torch.tensor([expectation_z])
        #result = torch.tensor(expectation_z)
        ctx.save_for_backward(input, result)

        return result

    @staticmethod
    def backward(ctx, grad_output):
        """ Backward pass computation """
        print('a')
        input, expectation_z = ctx.saved_tensors
        input_list = np.array(input.tolist())

        shift_right = input_list + np.ones(input_list.shape) * ctx.shift
        shift_left = input_list - np.ones(input_list.shape) * ctx.shift

        gradients = []
        for i in range(len(input_list)):
            expectation_right = ctx.quantum_circuit.run(shift_right[i])
            expectation_left  = ctx.quantum_circuit.run(shift_left[i])

            gradient = torch.tensor([expectation_right]) - torch.tensor([expectation_left])
            gradients.append(gradient)
        gradients = np.array([gradients]).T
        return torch.tensor([gradients]).float() * grad_output.float(), None, None

class Hybrid(nn.Module):
    """ Hybrid quantum - classical layer definition """

    def __init__(self, backend, shots, shift):
        super(Hybrid, self).__init__()
        self.quantum_circuit = QuantumCircuit(1, backend, shots)
        self.shift = shift

    def forward(self, input):
        #print('3',input)
        return HybridFunction.apply(input, self.quantum_circuit, self.shift)
```

```python
class LagrangianNeuralNetwork(torch.nn.Module):
    def __init__(self, D_in, hidden_list, D_out):
        """
        Neural Network used to approximate a paramaterized system lagrangian
        """
        super(LagrangianNeuralNetwork, self).__init__()
        self.model_layers = torch.nn.ModuleList()

        # input layer
        self.model_layers.append(torch.nn.Linear(D_in, hidden_list[0]))
        self.model_layers.append(torch.nn.Softplus())
        # add all hiden layers
        for i in range(1, len(hidden_list)):
            self.model_layers.append(torch.nn.Linear(hidden_list[i-1], hidden_list[i]))
            self.model_layers.append(torch.nn.Softplus())

        # output layer
        self.model_layers.append(torch.nn.Linear(hidden_list[-1], D_out))
        self.model_layers.append(torch.nn.Softplus())

    def forward(self, x):
        """
        applies all of the model layers, and returns the single output value,
        which in this case is the lagrangian of the system, representing the
        total energy
        """
        for layer in self.model_layers:
            x = layer(x)
        return x
```

```python
# determine device
print("--- Checking for CUDA Device... ---")
use_cuda = torch.cuda.is_available()
device = torch.device("cuda:0" if use_cuda else "cpu")
print(device)
```

    --- Checking for CUDA Device... ---
    cpu

```python
# organize data
input_size = train_inputs.shape[1]
output_size = 1  # for all lagrangian systems, output should be just a scalar energy value

# build model
print("--- Constructing Model... ---")
D_in = input_size  # state size
# hidden_list = [D_in, 256, 256, 256, 256, 256]
hidden_list = [D_in, 32, 64, 128, 256, 512, 256, 128, 64, 32]
D_out = output_size
lnn_model = LagrangianNeuralNetwork(D_in, hidden_list, D_out)
#summary(lnn_model, (1, 28, 28), device='cpu')
# set up training parameters
learning_rate = 1e-4
weight_decay = 1e-5
momentum = 0.9
num_epochs = 20
optimizer = torch.optim.Adam(lnn_model.parameters(),
                             lr=learning_rate,
                             weight_decay=weight_decay)

if os.path.isfile("model_weights.pth"):
    print("Re-loading existing weights!")
    checkpoint = torch.load("model_weights.pth")
    lnn_model.load_state_dict(checkpoint['model_state_dict'])
    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
```

    --- Constructing Model... ---

```python
# ensure model is in train mode so gradients are properly calculated
lnn_model.train()
# load device to either GPU or CPU depending on hardware
lnn_model.to(device)

# set up loss function
loss_fcn = torch.nn.MSELoss()

# set up GradScaler to improve run speed
scaler = torch.cuda.amp.GradScaler()

print("--- Beginning Training! ---")
loss_list = []
lnn_model.train()
for epoch in range(num_epochs):
    total_loss = []
    for batch_idx, (data, target) in enumerate(train_dataloader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        for p in lnn_model.parameters(): p.grad = None

        #Quantum layer for data encoding
        backend = qiskit.Aer.get_backend('qasm_simulator')
        num_qubit = 4
        circ = QCircuit(num_qubit, backend, 100)

        #Encoding data and target
        dataInput = torch.squeeze(torch.tensor(circ.run(data)))
        targetData = torch.tensor(circ.run(target))

        with torch.cuda.amp.autocast():
            target_pred = solve_euler_lagrange(lnn_model.forward, dataInput.float())
            loss = loss_fcn(target_pred.unsqueeze(0), targetData.float())

        # Backward pass
        scaler.scale(loss).backward()

        # Optimize the weights
        scaler.step(optimizer)

        # update the scale for next iteration
        scaler.update()

        total_loss.append(loss.item())

    loss_list.append(sum(total_loss)/len(total_loss))
    print('Training [{:.0f}%]\tLoss: {:.4f}'.format(100. * (epoch + 1) / num_epochs, loss_list[-1]))

print('end')
```

    --- Beginning Training! ---

    /usr/local/lib/python3.7/dist-packages/torch/cuda/amp/grad_scaler.py:115: UserWarning: torch.cuda.amp.GradScaler is enabled, but CUDA is not available.  Disabling.
      warnings.warn("torch.cuda.amp.GradScaler is enabled, but CUDA is not available.  Disabling.")
    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)
    /usr/local/lib/python3.7/dist-packages/torch/autocast_mode.py:141: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
      warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')
    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py:520: UserWarning: Using a target size (torch.Size([4])) that is different to the input size (torch.Size([1, 4])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
      return F.mse_loss(input, target, reduction=self.reduction)

    Training [5%]	Loss: 0.9506

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [10%]	Loss: 0.0107

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [15%]	Loss: 0.0101

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [20%]	Loss: 0.0098

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [25%]	Loss: 0.0094

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [30%]	Loss: 0.0100

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [35%]	Loss: 0.0105

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [40%]	Loss: 0.0100

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [45%]	Loss: 0.0102

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [50%]	Loss: 0.0099

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [55%]	Loss: 0.0095

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [60%]	Loss: 0.0099

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [65%]	Loss: 0.0098

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [70%]	Loss: 0.0090

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [75%]	Loss: 0.0094

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [80%]	Loss: 0.0090

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [85%]	Loss: 0.0091

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [90%]	Loss: 0.0088

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [95%]	Loss: 0.0087

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    Training [100%]	Loss: 0.0086
    end

```python
plt.plot(loss_list)
plt.title('Hybrid LQDNN Quantum-Classical Training (Four qubit)')
plt.xlabel('Training Iterations')
plt.ylabel('Loss')
```

```python
print("--- Beginning Test! ---")
lnn_model.eval()
total_lossTestHybridQC = []
correctTestQC = 0
for batch_idx, (data, target) in enumerate(test_dataloader):
    data, target = data.to(device), target.to(device)

    dataTest = torch.squeeze(torch.tensor(circ.run(data)))
    targetTest = torch.tensor(circ.run(target))

    with torch.cuda.amp.autocast():
        target_pred = solve_euler_lagrange(lnn_model.forward, dataTest.float())
        correctTestQC += target_pred.eq(targetTest.view_as(target_pred)).sum().item()
        testLoss = loss_fcn(target_pred.unsqueeze(0), targetTest.float())
    total_lossTestHybridQC.append(testLoss.item())

batch_size = 1
print('Performance on test data:\tLoss: {:.4f}\tAccuracy: {:.1f}%'.format(
    sum(total_lossTestHybridQC) / len(total_lossTestHybridQC),
    correctTestQC / len(test_dataloader) * 100 / batch_size)
    )

print('end')
```

    --- Beginning Test! ---

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)
    /usr/local/lib/python3.7/dist-packages/torch/autocast_mode.py:141: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
      warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')
    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py:520: UserWarning: Using a target size (torch.Size([4])) that is different to the input size (torch.Size([1, 4])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
      return F.mse_loss(input, target, reduction=self.reduction)

    Performance on test data:	Loss: 0.0085	Accuracy: 0.5%
    end

```python

```


```python
from pyvizml import CreateNBAData
import requests
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures
```

```python
# Initiation.
cnb = CreateNBAData(2019)
players = cnb.create_players_df()
```

    Creating players df...

```python
players
```

      firstName
      lastName
      temporaryDisplayName
      personId
      teamId
      jersey
      isActive
      pos
      heightFeet
      heightInches
      heightMeters
      weightPounds
      weightKilograms
      dateOfBirthUTC
      nbaDebutYear
      yearsPro
      collegeName
      lastAffiliation
      country
      isallStar

      0
      Jaylen
      Adams
      Adams, Jaylen
      1629121
      1610612757
      10
      True
      G
      6
      0
      1.83
      225
      102.1
      1996-05-04
      2018
      1
      St. Bonaventure
      St. Bonaventure/USA
      USA
      NaN

      1
      Steven
      Adams
      Adams, Steven
      203500
      1610612760
      12
      True
      C
      6
      11
      2.11
      265
      120.2
      1993-07-20
      2013
      6
      Pittsburgh
      Pittsburgh/New Zealand
      New Zealand
      NaN

      2
      Bam
      Adebayo
      Adebayo, Bam
      1628389
      1610612748
      13
      True
      C-F
      6
      9
      2.06
      255
      115.7
      1997-07-18
      2017
      2
      Kentucky
      Kentucky/USA
      USA
      NaN

      3
      LaMarcus
      Aldridge
      Aldridge, LaMarcus
      200746
      1610612759
      12
      True
      C-F
      6
      11
      2.11
      250
      113.4
      1985-07-19
      2006
      13
      Texas
      Texas/USA
      USA
      NaN

      4
      Kyle
      Alexander
      Alexander, Kyle
      1629734
      1610612748
      17
      True
      F-C
      6
      10
      2.08
      216
      98.0
      1996-10-21
      2019
      0
      Tennessee
      Tennessee/Canada
      Canada
      NaN

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      505
      Trae
      Young
      Young, Trae
      1629027
      1610612737
      11
      True
      G
      6
      1
      1.85
      180
      81.6
      1998-09-19
      2018
      1
      Oklahoma
      University of Oklahoma/USA
      USA
      NaN

      506
      Cody
      Zeller
      Zeller, Cody
      203469
      1610612766
      40
      True
      F-C
      7
      0
      2.13
      240
      108.9
      1992-10-05
      2013
      6
      Indiana
      Indiana/USA
      USA
      NaN

      507
      Tyler
      Zeller
      Zeller, Tyler
      203092
      1610612759
      40
      True
      F-C
      6
      11
      2.11
      250
      113.4
      1990-01-17
      2012
      7
      North Carolina
      North Carolina/USA
      USA
      NaN

      508
      Ante
      Zizic
      Zizic, Ante
      1627790
      1610612739
      41
      True
      C
      6
      10
      2.08
      266
      120.7
      1997-01-04
      2017
      2

      Darussafaka (Turkey)/Croatia
      Croatia
      NaN

      509
      Ivica
      Zubac
      Zubac, Ivica
      1627826
      1610612746
      40
      True
      C
      7
      0
      2.13
      240
      108.9
      1997-03-18
      2016
      3

      Mega Leks/Croatia
      Croatia
      NaN

510 rows × 20 columns

```python
X = players['heightMeters'].values
```

```python
X
```

    array(['1.83', '2.11', '2.06', '2.11', '2.08', '1.96', '1.93', '2.11',
           '2.03', '1.96', '2.06', '2.11', '2.08', '1.98', '2.03', '2.01',
           '1.9', '2.03', '1.8', '2.11', '1.98', '2.11', '1.98', '2.13',
           '1.78', '2.03', '1.98', '1.96', '2.03', '2.06', '2.08', '1.93',
           '2.03', '1.9', '1.93', '1.96', '2.03', '1.96', '2.08', '1.85',
           '2.06', '2.11', '2.03', '2.08', '1.85', '1.98', '2.03', '2.18',
           '1.9', '2.03', '1.96', '2.06', '1.98', '1.85', '1.9', '2.08',
           '1.96', '1.98', '2.06', '1.98', '1.98', '2.01', '1.98', '1.96',
           '2.01', '1.93', '1.98', '2.18', '1.96', '1.98', '1.98', '1.85',
           '2.08', '1.98', '1.83', '1.98', '1.93', '2.01', '2.06', '2.03',
           '1.96', '2.03', '2.08', '1.98', '1.85', '1.98', '2.06', '1.96',
           '1.96', '2.13', '2.13', '2.03', '1.96', '1.8', '2.06', '1.98',
           '2.03', '1.93', '2.11', '1.75', '1.96', '2.01', '2.06', '2.11',
           '1.85', '1.96', '1.85', '2.03', '2.01', '2.01', '1.96', '1.98',
           '1.98', '1.88', '1.9', '1.93', '2.08', '2.06', '1.93', '2.13',
           '1.9', '1.98', '2.03', '1.96', '2.08', '1.96', '1.93', '2.01',
           '1.9', '1.96', '2.03', '1.98', '1.9', '2.08', '1.98', '1.9',
           '2.08', '1.8', '1.93', '2.13', '1.98', '2.06', '1.93', '1.96',
           '2.26', '2.06', '2.08', '1.98', '2.06', '1.83', '2.01', '1.88',
           '2.01', '1.9', '1.9', '1.96', '1.9', '2.06', '2.08', '2.08',
           '1.85', '1.85', '2.11', '2.03', '2.03', '2.06', '2.11', '1.96',
           '2.16', '1.83', '2.03', '1.9', '1.85', '1.96', '2.03', '1.93',
           '1.83', '1.98', '1.98', '2.03', '1.93', '2.03', '2.06', '1.98',
           '1.85', '2.03', '1.96', '2.06', '1.96', '1.96', '2.01', '1.78',
           '2.01', '1.93', '1.98', '2.03', '2.01', '1.96', '2.03', '2.11',
           '2.01', '2.06', '2.08', '2.06', '2.11', '1.96', '2.06', '2.03',
           '1.93', '1.9', '1.98', '2.03', '1.83', '1.9', '1.98', '1.98',
           '2.08', '2.03', '2.06', '1.93', '1.98', '2.08', '2.03', '2.01',
           '2.01', '2.01', '2.13', '1.98', '2.06', '2.01', '2.01', '1.88',
           '2.11', '1.98', '1.9', '2.03', '2.01', '1.9', '2.11', '2.01',
           '2.06', '1.96', '1.96', '2.01', '2.01', '2.03', '2.01', '1.96',
           '1.98', '1.9', '2.13', '2.11', '1.83', '1.98', '2.11', '1.9',
           '2.06', '2.13', '2.08', '1.96', '1.98', '2.01', '2.08', '1.88',
           '2.01', '1.96', '2.01', '2.18', '2.01', '2.06', '2.03', '2.08',
           '1.96', '1.93', '1.98', '2.01', '2.03', '2.08', '1.93', '1.96',
           '1.96', '2.13', '2.01', '2.13', '1.98', '1.88', '1.96', '2.06',
           '2.13', '2.13', '2.03', '1.83', '2.01', '2.06', '2.11', '2.13',
           '1.96', '2.24', '2.13', '1.96', '1.96', '1.88', '1.96', '1.8',
           '1.96', '1.93', '2.03', '2.01', '1.9', '1.85', '2.08', '2.01',
           '2.13', '1.93', '2.01', '1.8', '1.9', '1.96', '2.06', '1.88',
           '2.06', '2.01', '1.98', '2.01', '1.85', '2.01', '1.96', '1.85',
           '1.9', '1.96', '1.9', '1.88', '1.9', '1.9', '2.01', '2.03', '1.88',
           '2.03', '2.03', '1.9', '1.9', '1.93', '1.93', '2.06', '2.08',
           '2.01', '1.96', '2.01', '1.83', '1.85', '2.01', '2.11', '2.08',
           '1.93', '1.93', '1.88', '2.13', '1.96', '1.93', '2.06', '1.98',
           '2.08', '1.9', '1.93', '2.03', '1.93', '2.11', '1.96', '2.01',
           '2.01', '2.08', '2.03', '1.98', '2.18', '2.03', '2.11', '1.85',
           '1.85', '1.9', '1.9', '2.08', '1.96', '2.11', '2.16', '2.13',
           '1.93', '2.11', '1.93', '2.08', '2.03', '2.08', '2.21', '2.08',
           '1.9', '2.01', '2.03', '1.93', '2.03', '1.9', '2.06', '2.01',
           '1.96', '1.9', '2.01', '2.01', '1.93', '2.13', '1.98', '2.03',
           '1.85', '1.88', '1.98', '1.85', '1.9', '1.93', '2.11', '2.08',
           '2.01', '2.08', '2.01', '1.96', '1.85', '2.01', '1.98', '1.85',
           '1.93', '1.96', '2.06', '2.03', '2.08', '1.96', '1.9', '2.08',
           '1.9', '1.83', '1.98', '1.9', '1.88', '1.98', '2.06', '2.03',
           '1.96', '1.93', '2.06', '2.03', '1.9', '1.96', '2.03', '1.9',
           '2.03', '1.93', '1.98', '2.06', '1.93', '1.96', '2.03', '1.98',
           '2.11', '1.96', '1.96', '1.9', '1.98', '2.11', '2.06', '2.11',
           '1.93', '2.06', '1.85', '1.9', '2.08', '2.11', '2.06', '2.11',
           '1.9', '1.83', '1.96', '1.93', '1.9', '2.03', '2.01', '2.03',
           '1.78', '1.98', '1.9', '1.9', '1.93', '1.93', '2.13', '2.01',
           '1.98', '2.06', '1.98', '1.85', '2.03', '1.88', '2.03', '1.98',
           '2.08', '1.98', '1.98', '2.08', '2.03', '1.96', '1.83', '2.03',
           '1.85', '2.13', '2.11', '2.08', '2.13'], dtype=object)

```python
X = players['heightMeters'].values.astype(float)
```

```python
X
```

    array([1.83, 2.11, 2.06, 2.11, 2.08, 1.96, 1.93, 2.11, 2.03, 1.96, 2.06,
           2.11, 2.08, 1.98, 2.03, 2.01, 1.9 , 2.03, 1.8 , 2.11, 1.98, 2.11,
           1.98, 2.13, 1.78, 2.03, 1.98, 1.96, 2.03, 2.06, 2.08, 1.93, 2.03,
           1.9 , 1.93, 1.96, 2.03, 1.96, 2.08, 1.85, 2.06, 2.11, 2.03, 2.08,
           1.85, 1.98, 2.03, 2.18, 1.9 , 2.03, 1.96, 2.06, 1.98, 1.85, 1.9 ,
           2.08, 1.96, 1.98, 2.06, 1.98, 1.98, 2.01, 1.98, 1.96, 2.01, 1.93,
           1.98, 2.18, 1.96, 1.98, 1.98, 1.85, 2.08, 1.98, 1.83, 1.98, 1.93,
           2.01, 2.06, 2.03, 1.96, 2.03, 2.08, 1.98, 1.85, 1.98, 2.06, 1.96,
           1.96, 2.13, 2.13, 2.03, 1.96, 1.8 , 2.06, 1.98, 2.03, 1.93, 2.11,
           1.75, 1.96, 2.01, 2.06, 2.11, 1.85, 1.96, 1.85, 2.03, 2.01, 2.01,
           1.96, 1.98, 1.98, 1.88, 1.9 , 1.93, 2.08, 2.06, 1.93, 2.13, 1.9 ,
           1.98, 2.03, 1.96, 2.08, 1.96, 1.93, 2.01, 1.9 , 1.96, 2.03, 1.98,
           1.9 , 2.08, 1.98, 1.9 , 2.08, 1.8 , 1.93, 2.13, 1.98, 2.06, 1.93,
           1.96, 2.26, 2.06, 2.08, 1.98, 2.06, 1.83, 2.01, 1.88, 2.01, 1.9 ,
           1.9 , 1.96, 1.9 , 2.06, 2.08, 2.08, 1.85, 1.85, 2.11, 2.03, 2.03,
           2.06, 2.11, 1.96, 2.16, 1.83, 2.03, 1.9 , 1.85, 1.96, 2.03, 1.93,
           1.83, 1.98, 1.98, 2.03, 1.93, 2.03, 2.06, 1.98, 1.85, 2.03, 1.96,
           2.06, 1.96, 1.96, 2.01, 1.78, 2.01, 1.93, 1.98, 2.03, 2.01, 1.96,
           2.03, 2.11, 2.01, 2.06, 2.08, 2.06, 2.11, 1.96, 2.06, 2.03, 1.93,
           1.9 , 1.98, 2.03, 1.83, 1.9 , 1.98, 1.98, 2.08, 2.03, 2.06, 1.93,
           1.98, 2.08, 2.03, 2.01, 2.01, 2.01, 2.13, 1.98, 2.06, 2.01, 2.01,
           1.88, 2.11, 1.98, 1.9 , 2.03, 2.01, 1.9 , 2.11, 2.01, 2.06, 1.96,
           1.96, 2.01, 2.01, 2.03, 2.01, 1.96, 1.98, 1.9 , 2.13, 2.11, 1.83,
           1.98, 2.11, 1.9 , 2.06, 2.13, 2.08, 1.96, 1.98, 2.01, 2.08, 1.88,
           2.01, 1.96, 2.01, 2.18, 2.01, 2.06, 2.03, 2.08, 1.96, 1.93, 1.98,
           2.01, 2.03, 2.08, 1.93, 1.96, 1.96, 2.13, 2.01, 2.13, 1.98, 1.88,
           1.96, 2.06, 2.13, 2.13, 2.03, 1.83, 2.01, 2.06, 2.11, 2.13, 1.96,
           2.24, 2.13, 1.96, 1.96, 1.88, 1.96, 1.8 , 1.96, 1.93, 2.03, 2.01,
           1.9 , 1.85, 2.08, 2.01, 2.13, 1.93, 2.01, 1.8 , 1.9 , 1.96, 2.06,
           1.88, 2.06, 2.01, 1.98, 2.01, 1.85, 2.01, 1.96, 1.85, 1.9 , 1.96,
           1.9 , 1.88, 1.9 , 1.9 , 2.01, 2.03, 1.88, 2.03, 2.03, 1.9 , 1.9 ,
           1.93, 1.93, 2.06, 2.08, 2.01, 1.96, 2.01, 1.83, 1.85, 2.01, 2.11,
           2.08, 1.93, 1.93, 1.88, 2.13, 1.96, 1.93, 2.06, 1.98, 2.08, 1.9 ,
           1.93, 2.03, 1.93, 2.11, 1.96, 2.01, 2.01, 2.08, 2.03, 1.98, 2.18,
           2.03, 2.11, 1.85, 1.85, 1.9 , 1.9 , 2.08, 1.96, 2.11, 2.16, 2.13,
           1.93, 2.11, 1.93, 2.08, 2.03, 2.08, 2.21, 2.08, 1.9 , 2.01, 2.03,
           1.93, 2.03, 1.9 , 2.06, 2.01, 1.96, 1.9 , 2.01, 2.01, 1.93, 2.13,
           1.98, 2.03, 1.85, 1.88, 1.98, 1.85, 1.9 , 1.93, 2.11, 2.08, 2.01,
           2.08, 2.01, 1.96, 1.85, 2.01, 1.98, 1.85, 1.93, 1.96, 2.06, 2.03,
           2.08, 1.96, 1.9 , 2.08, 1.9 , 1.83, 1.98, 1.9 , 1.88, 1.98, 2.06,
           2.03, 1.96, 1.93, 2.06, 2.03, 1.9 , 1.96, 2.03, 1.9 , 2.03, 1.93,
           1.98, 2.06, 1.93, 1.96, 2.03, 1.98, 2.11, 1.96, 1.96, 1.9 , 1.98,
           2.11, 2.06, 2.11, 1.93, 2.06, 1.85, 1.9 , 2.08, 2.11, 2.06, 2.11,
           1.9 , 1.83, 1.96, 1.93, 1.9 , 2.03, 2.01, 2.03, 1.78, 1.98, 1.9 ,
           1.9 , 1.93, 1.93, 2.13, 2.01, 1.98, 2.06, 1.98, 1.85, 2.03, 1.88,
           2.03, 1.98, 2.08, 1.98, 1.98, 2.08, 2.03, 1.96, 1.83, 2.03, 1.85,
           2.13, 2.11, 2.08, 2.13])

```python
X = players['heightMeters'].values.astype(float).reshape(-1, 1)
```

```python
X
```

    array([[1.83],
           [2.11],
           [2.06],
           [2.11],
           [2.08],
           [1.96],
           [1.93],
           [2.11],
           [2.03],
           [1.96],
           [2.06],
           [2.11],
           [2.08],
           [1.98],
           [2.03],
           [2.01],
           [1.9 ],
           [2.03],
           [1.8 ],
           [2.11],
           [1.98],
           [2.11],
           [1.98],
           [2.13],
           [1.78],
           [2.03],
           [1.98],
           [1.96],
           [2.03],
           [2.06],
           [2.08],
           [1.93],
           [2.03],
           [1.9 ],
           [1.93],
           [1.96],
           [2.03],
           [1.96],
           [2.08],
           [1.85],
           [2.06],
           [2.11],
           [2.03],
           [2.08],
           [1.85],
           [1.98],
           [2.03],
           [2.18],
           [1.9 ],
           [2.03],
           [1.96],
           [2.06],
           [1.98],
           [1.85],
           [1.9 ],
           [2.08],
           [1.96],
           [1.98],
           [2.06],
           [1.98],
           [1.98],
           [2.01],
           [1.98],
           [1.96],
           [2.01],
           [1.93],
           [1.98],
           [2.18],
           [1.96],
           [1.98],
           [1.98],
           [1.85],
           [2.08],
           [1.98],
           [1.83],
           [1.98],
           [1.93],
           [2.01],
           [2.06],
           [2.03],
           [1.96],
           [2.03],
           [2.08],
           [1.98],
           [1.85],
           [1.98],
           [2.06],
           [1.96],
           [1.96],
           [2.13],
           [2.13],
           [2.03],
           [1.96],
           [1.8 ],
           [2.06],
           [1.98],
           [2.03],
           [1.93],
           [2.11],
           [1.75],
           [1.96],
           [2.01],
           [2.06],
           [2.11],
           [1.85],
           [1.96],
           [1.85],
           [2.03],
           [2.01],
           [2.01],
           [1.96],
           [1.98],
           [1.98],
           [1.88],
           [1.9 ],
           [1.93],
           [2.08],
           [2.06],
           [1.93],
           [2.13],
           [1.9 ],
           [1.98],
           [2.03],
           [1.96],
           [2.08],
           [1.96],
           [1.93],
           [2.01],
           [1.9 ],
           [1.96],
           [2.03],
           [1.98],
           [1.9 ],
           [2.08],
           [1.98],
           [1.9 ],
           [2.08],
           [1.8 ],
           [1.93],
           [2.13],
           [1.98],
           [2.06],
           [1.93],
           [1.96],
           [2.26],
           [2.06],
           [2.08],
           [1.98],
           [2.06],
           [1.83],
           [2.01],
           [1.88],
           [2.01],
           [1.9 ],
           [1.9 ],
           [1.96],
           [1.9 ],
           [2.06],
           [2.08],
           [2.08],
           [1.85],
           [1.85],
           [2.11],
           [2.03],
           [2.03],
           [2.06],
           [2.11],
           [1.96],
           [2.16],
           [1.83],
           [2.03],
           [1.9 ],
           [1.85],
           [1.96],
           [2.03],
           [1.93],
           [1.83],
           [1.98],
           [1.98],
           [2.03],
           [1.93],
           [2.03],
           [2.06],
           [1.98],
           [1.85],
           [2.03],
           [1.96],
           [2.06],
           [1.96],
           [1.96],
           [2.01],
           [1.78],
           [2.01],
           [1.93],
           [1.98],
           [2.03],
           [2.01],
           [1.96],
           [2.03],
           [2.11],
           [2.01],
           [2.06],
           [2.08],
           [2.06],
           [2.11],
           [1.96],
           [2.06],
           [2.03],
           [1.93],
           [1.9 ],
           [1.98],
           [2.03],
           [1.83],
           [1.9 ],
           [1.98],
           [1.98],
           [2.08],
           [2.03],
           [2.06],
           [1.93],
           [1.98],
           [2.08],
           [2.03],
           [2.01],
           [2.01],
           [2.01],
           [2.13],
           [1.98],
           [2.06],
           [2.01],
           [2.01],
           [1.88],
           [2.11],
           [1.98],
           [1.9 ],
           [2.03],
           [2.01],
           [1.9 ],
           [2.11],
           [2.01],
           [2.06],
           [1.96],
           [1.96],
           [2.01],
           [2.01],
           [2.03],
           [2.01],
           [1.96],
           [1.98],
           [1.9 ],
           [2.13],
           [2.11],
           [1.83],
           [1.98],
           [2.11],
           [1.9 ],
           [2.06],
           [2.13],
           [2.08],
           [1.96],
           [1.98],
           [2.01],
           [2.08],
           [1.88],
           [2.01],
           [1.96],
           [2.01],
           [2.18],
           [2.01],
           [2.06],
           [2.03],
           [2.08],
           [1.96],
           [1.93],
           [1.98],
           [2.01],
           [2.03],
           [2.08],
           [1.93],
           [1.96],
           [1.96],
           [2.13],
           [2.01],
           [2.13],
           [1.98],
           [1.88],
           [1.96],
           [2.06],
           [2.13],
           [2.13],
           [2.03],
           [1.83],
           [2.01],
           [2.06],
           [2.11],
           [2.13],
           [1.96],
           [2.24],
           [2.13],
           [1.96],
           [1.96],
           [1.88],
           [1.96],
           [1.8 ],
           [1.96],
           [1.93],
           [2.03],
           [2.01],
           [1.9 ],
           [1.85],
           [2.08],
           [2.01],
           [2.13],
           [1.93],
           [2.01],
           [1.8 ],
           [1.9 ],
           [1.96],
           [2.06],
           [1.88],
           [2.06],
           [2.01],
           [1.98],
           [2.01],
           [1.85],
           [2.01],
           [1.96],
           [1.85],
           [1.9 ],
           [1.96],
           [1.9 ],
           [1.88],
           [1.9 ],
           [1.9 ],
           [2.01],
           [2.03],
           [1.88],
           [2.03],
           [2.03],
           [1.9 ],
           [1.9 ],
           [1.93],
           [1.93],
           [2.06],
           [2.08],
           [2.01],
           [1.96],
           [2.01],
           [1.83],
           [1.85],
           [2.01],
           [2.11],
           [2.08],
           [1.93],
           [1.93],
           [1.88],
           [2.13],
           [1.96],
           [1.93],
           [2.06],
           [1.98],
           [2.08],
           [1.9 ],
           [1.93],
           [2.03],
           [1.93],
           [2.11],
           [1.96],
           [2.01],
           [2.01],
           [2.08],
           [2.03],
           [1.98],
           [2.18],
           [2.03],
           [2.11],
           [1.85],
           [1.85],
           [1.9 ],
           [1.9 ],
           [2.08],
           [1.96],
           [2.11],
           [2.16],
           [2.13],
           [1.93],
           [2.11],
           [1.93],
           [2.08],
           [2.03],
           [2.08],
           [2.21],
           [2.08],
           [1.9 ],
           [2.01],
           [2.03],
           [1.93],
           [2.03],
           [1.9 ],
           [2.06],
           [2.01],
           [1.96],
           [1.9 ],
           [2.01],
           [2.01],
           [1.93],
           [2.13],
           [1.98],
           [2.03],
           [1.85],
           [1.88],
           [1.98],
           [1.85],
           [1.9 ],
           [1.93],
           [2.11],
           [2.08],
           [2.01],
           [2.08],
           [2.01],
           [1.96],
           [1.85],
           [2.01],
           [1.98],
           [1.85],
           [1.93],
           [1.96],
           [2.06],
           [2.03],
           [2.08],
           [1.96],
           [1.9 ],
           [2.08],
           [1.9 ],
           [1.83],
           [1.98],
           [1.9 ],
           [1.88],
           [1.98],
           [2.06],
           [2.03],
           [1.96],
           [1.93],
           [2.06],
           [2.03],
           [1.9 ],
           [1.96],
           [2.03],
           [1.9 ],
           [2.03],
           [1.93],
           [1.98],
           [2.06],
           [1.93],
           [1.96],
           [2.03],
           [1.98],
           [2.11],
           [1.96],
           [1.96],
           [1.9 ],
           [1.98],
           [2.11],
           [2.06],
           [2.11],
           [1.93],
           [2.06],
           [1.85],
           [1.9 ],
           [2.08],
           [2.11],
           [2.06],
           [2.11],
           [1.9 ],
           [1.83],
           [1.96],
           [1.93],
           [1.9 ],
           [2.03],
           [2.01],
           [2.03],
           [1.78],
           [1.98],
           [1.9 ],
           [1.9 ],
           [1.93],
           [1.93],
           [2.13],
           [2.01],
           [1.98],
           [2.06],
           [1.98],
           [1.85],
           [2.03],
           [1.88],
           [2.03],
           [1.98],
           [2.08],
           [1.98],
           [1.98],
           [2.08],
           [2.03],
           [1.96],
           [1.83],
           [2.03],
           [1.85],
           [2.13],
           [2.11],
           [2.08],
           [2.13]])

```python
X = players['heightMeters'].values.astype(float).reshape(-1, 1)
y = players['weightKilograms'].values.astype(float)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.33, random_state=42) # random_state => shuffle
h = LinearRegression()
# h = LinearRegression(fir_intercept=False)
h.fit(X_train, y_train)
print(h.coef_)
print(h.intercept_)
h.predict(X_valid)[:10]
```

    [96.22270241]
    -93.22909204641577

    array([102.10299386, 102.10299386,  95.36740469,  82.85845337,
           100.17853981, 106.91412898,  97.29185874,  92.48072361,
           102.10299386, 100.17853981])

```python

```

```python
# Wrtie a class to simulate LinearRegression() function.
```

```python
class NormalEquation:
    def __init__(self, fit_intercept=True):
        self._fit_intercept = fit_intercept
    def fit(self, X_train, y_train):
        self._X_train = X_train.copy() # Not affect original data.
        self._y_train = y_train.copy() # Not affect original data.
        m = self._X_train.shape[0]

        # Determine add intercept or not.
        if self._fit_intercept:
            X0 = np.ones(m).reshape(-1, 1)
            self._X_train = np.concatenate([X0, self._X_train], axis=1) # axis=1 => row
        X_train_T = np.transpose(self._X_train)
        left_matrix = np.dot(X_train_T, self._X_train)  # (2, m)(m, 2) = (2, 2)
        right_matrix = np.dot(X_train_T, self._y_train) # (2, m)(m,) = (2,)
        left_matrix_inv = np.linalg.inv(left_matrix)
        w = np.dot(left_matrix_inv, right_matrix)       # (2, 2)(2,) = (2,)
        self._w = w
        self.intercept_ = w[0]
        self.coef_ = w[1:]
    def predict(self, X):
        m = X.shape[0]
        if self._fit_intercept:
            X0 = np.ones(m).reshape(-1, 1)
            X = np.concatenate([X0, X], axis=1)
        y_pred = np.dot(X, self._w) # (m, 2)(2, ) = (m,)
        return y_pred
```

```python
h = NormalEquation()
h.fit(X_train, y_train)
```

```python
h._X_train.shape
```

    (341, 2)

```python
h._X_train[:5]
```

    array([[1.  , 1.85],
           [1.  , 2.24],
           [1.  , 2.06],
           [1.  , 2.01],
           [1.  , 2.11]])

```python
print(h.intercept_)
print(h.coef_)
```

    -93.22909204637108
    [96.22270241]

```python
h.predict(X_valid)[:10]
```

    array([102.10299386, 102.10299386,  95.36740469,  82.85845337,
           100.17853981, 106.91412898,  97.29185874,  92.48072361,
           102.10299386, 100.17853981])

\begin{equation}
w^* = (X^TX)^{-1}X^Ty
\end{equation}


```python
%matplotlib inline
```

## Intro

Este tutorial é baseado no tutorial [Aprendendo representações de frases utilizando um codificador-decodificador para máquinas de tradução estatística](https://github.com/bentrevett/pytorch-seq2seq/blob/master/2%20-%20Learning%20Phrase%20Representations%20using%20RNN%20Encoder-Decoder%20for%20Statistical%20Machine%20Translation.ipynb)

### Dependências

* torchtext
* spacy

### Arquitetura de um codificador-decodificador

Vamos lembrar da visão geral de um codificador-decodificador:

Nós usamos o codificador (em verde) na sequência de entrada para gerar um vetor de contexto `z` (em vermelho).
Esse vetor é então utilizado em um decodificador (em azul) e uma camada linear (em roxo) para gerar a sequência de saída.

Neste modelo, estamos usando um modelo de múltiplas camadas implementando uma memória de curto e longo prazo (`LSTM`):

Um dos problemas deste modelo, é que o decodificador está tentando colocar muita informação nos estados intermediários do nosso modelo.  No momento da decodificação, o estado intermediário deverá conter informação sobre toda a sequência de entrada codificada até o momento bem como todos os tokens decodificados até então. Isto exige muita memória. Seria interessante amenizar o processo de compressão para que possamos ter um modelo melhor!

Para isso, usaremos uma unidade recorrente de porta (`GRU`).

## Dados

Inicialmente, vamos importar algumas das bibliotecas necessárias para manipular os nossos dados:

```python
import torch
import torch.nn as nn
import torch.optim as optim

from torchtext.datasets import TranslationDataset, Multi30k
from torchtext.data import Field, BucketIterator

import spacy

import random
import math
import os
import time
```

Em seguida, vamos usar a mesma `SEED` para garantir que os nossos resultados sejam reproduzíveis/determinísticos

```python
SEED = 1

random.seed(SEED)
torch.manual_seed(SEED)
torch.backends.cudnn.deterministic = True
```

Por fim, vamos utilizar modelos em alemão e inglês.

**obs.:** Eu gostaria muito de que este tutorial fosse em alemão/português ou inglês/português, mas eu preciso achar um módulo/base de dados com ambos os idiomas

No seu ambiente conda, execute a linha de comando abaixo para baixar os modelos:

```bash
python -m spacy download en
python -m spacy download de
```

Em caso de sucesso, você obterá uma mensagem similar a:

```bash
    You can now load the model via spacy.load('en')
```

```python
spacy_de = spacy.load('de')
spacy_en = spacy.load('en')
```

Ao processar os textos de entrada, vamos utilizar uma técnica chamada de [tokenização](https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization). Basicamente, teremos uma sentença de entrada e fragmentaremos a sentence em unidades léxicas, como por exemplo, palavras.

```python
def tokenize_de(text):
    """
    Tokenizes German text from a string into a list of strings
    """
    return [tok.text for tok in spacy_de.tokenizer(text)]

def tokenize_en(text):
    """
    Tokenizes English text from a string into a list of strings
    """
    return [tok.text for tok in spacy_en.tokenizer(text)]
```

Também vamos definir dois tokens especiais para indicar o início de uma sentença (`sos`) e o final de uma sentença (`eos`).
Também converteremos a sentença para que ela contenha apenas letras minúsculas.

```python
SRC = Field(tokenize=tokenize_de, init_token='<sos>', eos_token='<eos>', lower=True)
TRG = Field(tokenize=tokenize_en, init_token='<sos>', eos_token='<eos>', lower=True)
```

Finalmente, vamos carregar nossos dados:

```python
train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'), fields=(SRC, TRG))
```

    downloading training.tar.gz

    training.tar.gz: 100%|██████████| 1.21M/1.21M [00:03<00:00, 352kB/s]

    downloading validation.tar.gz

    validation.tar.gz: 100%|██████████| 46.3k/46.3k [00:00<00:00, 143kB/s]

    downloading mmt_task1_test2016.tar.gz

    mmt_task1_test2016.tar.gz: 100%|██████████| 66.2k/66.2k [00:00<00:00, 128kB/s]

Vamos checar uma frase de entrada e saída para garantir que nossos dados estão corretos:

```python
import pprint

pp = pprint.PrettyPrinter(indent=4)
pp.pprint(vars(train_data.examples[0]))
```

    {   'src': [   'zwei',
                   'junge',
                   'weiße',
                   'männer',
                   'sind',
                   'im',
                   'freien',
                   'in',
                   'der',
                   'nähe',
                   'vieler',
                   'büsche',
                   '.'],
        'trg': [   'two',
                   'young',
                   ',',
                   'white',
                   'males',
                   'are',
                   'outside',
                   'near',
                   'many',
                   'bushes',
                   '.']}

Por fim, vamos construir o nosso vocabulário convertendo todas as palavras que aparecem menos de duas vezes em termos desconhecidos (`<unk>`)

```python
SRC.build_vocab(train_data, min_freq=2)
TRG.build_vocab(train_data, min_freq=2)
```

Também vamos dividir os nossos dados em dados de treinamento, validação e testes.

A primeira linha define se pytorch utilizará uma unidade de processamento `CUDA` ou uma `CPU`

```python
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

BATCH_SIZE = 128

train_iterator, valid_iterator, test_iterator = BucketIterator.splits(
    (train_data, valid_data, test_data), batch_size=BATCH_SIZE, device=device)
```

## Construindo o nosso modelo

### Codificador

O codificador será um modelo com uma única camada de memória de curto longo prazo (`multi-layer LSTM`). E utilizaremos uma única unidade de porta recorrente. Não passaremos valores de dropout para a GRU uma vez que o dropout é utilizado entre as camadas de uma RNN. Como temos apenas uma única camada, PyTorch PyTorch alertará que temos uma única camada no nosso modelo.

Outra informação importante sobre uma GRU é que ela requer e retorna uma única camada intermediária. Não há um estado como em uma LSTM.

\begin{equation}
h_t = \text{GRU}(x_t, h_{t - 1}) \\
(h_t, c_t) = \text{LSTM}(x_t, (h_{t - 1}, c_{t - 1})) \\
h_t = \text{RNN}(x_t, h_{t - 1})
\end{equation}

De acordo com a equação acima, não há tanta diferença entre uma GRU e uma RNN. Entretanto, dentro de uma GRU existem inúmeros mecânismos de porta que controlam o fluxo de informação entrando e saindo de um estado intermediário. Para mais detalhes, checar este [post](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) (em inglês).

**obs.:** No futuro, eu pretendo fazer um tutorial explicando em mais detalhes como uma LSTM funciona.

O restante do codificador segue a arquitetura "padrão" de um codificador. Isto é, o codificador recebe a sequência de entrada $X = {x_1, x_2, ..., x_T}$ e computa estados intermediários de forma recorrende, $H = {h_1, h_2, ..., h_T}$. Por fim, é retornado o vetor de contexto (que corresponde ao último estado intermediário computado), $z = h_T$. Onde

\begin{equation}
h_t = \text{EncoderGRU}(x_t, h_{t - 1})
\end{equation}

Este codificador é idêntico a arquitetura de um codificador de um modelo de sequência para sequência (`seq2seq`), mas toda a "mágica" ocorre dentro da GRU (em verde)

```python
class Encoder(nn.Module):
    def __init__(self, input_dim, emb_dim, hid_dim, dropout):
        super().__init__()

        self.input_dim = input_dim
        self.emb_dim = emb_dim
        self.hid_dim = hid_dim
        self.dropout = dropout

        self.embedding = nn.Embedding(input_dim, emb_dim) #no dropout as only one layer!

        self.rnn = nn.GRU(emb_dim, hid_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, src):

        #src = [src sent len, batch size]

        embedded = self.dropout(self.embedding(src))

        #embedded = [src sent len, batch size, emb dim]

        outputs, hidden = self.rnn(embedded) #no cell state!

        #outputs = [src sent len, batch size, hid dim * n directions]
        #hidden = [n layers * n directions, batch size, hid dim]

        #outputs are always from the top hidden layer

        return hidden
```

### Decodificador

O decodificador é onde tentamos aliviar o problema da compressão de informação (discutido na introdução deste notebook).

Ao invés da GRU no decodificador utilizar apenas as palavras de saída $y_t$ e o estado intermediário anterior $s_{t - 1}$ como entrada, o decodificador também utilizará o vetor de contexto $Z$

\begin{equation}
s_t = \text{DecoderGRU}(y_t, h_{t - 1}, Z)
\end{equation}

Note que o vetor de contexto $Z$ não possui um parâmetro $t$. Iso indica que utilizamos o mesmo vetor de contexto (retornado pelo codificador) a cada intervalo de tempo $t$.

Para predizermos o próximo estado, utilizaremos uma camada linear, $f$. Esta camada utiliza apenas a última camada intermediária do decodificador naquele intervalo de tempo, $s_t$. Sendo assim, para predizermos $\hat{y}_{t + 1}$, passaremos como parâmetros o token atual na sequência $\hat{y_t}$ e o vetor de contexto para uma camada linear, conforme equação:

\begin{equation}
\hat{y}_{t + 1} = f(y_t, s_t, Z)
\end{equation}

O nosso decodificador se parece com algo como:

O estado intermediário inicial, $s_0$ é o vetor de contexto $Z$. Então, ao gerar o primeiro token da saída, nos estamos passando dois vetores indênticos para a GRU.

**Como essas modificações reduzem a compressão de informação?** Tecnicamente, o estado intermediário $s_t$ não mais precisa manter qualquer informação sobre a sequência de entrada, uma vez que isto esta disponível como uma entrada através do vetor de contexto. Sendo assim, ele só precisa manter informação sobre os tokens gerados até então. A adição de $y_t$ a camanda linear também significa que esta camada pode utilizar qual é o último token visto, sem necessariamente necessitar desta informação como algo que deveria estar comprimido no último estado intermediário computado.

Entretanto, a explicação anterior é apenas uma hipótese. É impossível determinar com precisão como o decodificador utiliza toda a informação fornecida. Dito isto, a explicação anterior serve como uma boa intuição para compreender o que está acontecendo e os resultados oriundos destas modificações indicam que estas modificações são uma boa ideia!

**Implementação** passaremos $y_t$ e o vetor de contexto $Z$ para a GRU concatenando ambos os vetores. Desta forma, as dimensões do vetor de entrada da GRU serão *emb_dim + hid_dim*. A camada linear tomará como entrada $y_t$, $s_t$ e $Z$ concatenando estes vetores. Assim, o vetor de entrada para a camada linear terá dimensão *emb_dim + 2 x hid_dim*. Também não utilizaremos um dropout uma vez que a GRU só possui uma única camada.

A função *forward* agora receberá um parâmetro, i.e. o contexto. Dentro da função, concatenaremos $y_t$ e $z$ em *emb_con* antes de passar este vetor para a GRU. Também concatenamos $y_t$, $s_t$ e $Z$ e passamos estes vetores para a camada linear para obtermos uma predição, isto é, $\hat{y}_{t + 1}$.

```python
class Decoder(nn.Module):
    def __init__(self, output_dim, emb_dim, hid_dim, dropout):
        super().__init__()

        self.emb_dim = emb_dim
        self.hid_dim = hid_dim
        self.output_dim = output_dim
        self.dropout = dropout

        self.embedding = nn.Embedding(output_dim, emb_dim)

        self.rnn = nn.GRU(emb_dim + hid_dim, hid_dim)

        self.out = nn.Linear(emb_dim + hid_dim*2, output_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, input, hidden, context):

        #input = [batch size]
        #hidden = [n layers * n directions, batch size, hid dim]
        #context = [n layers * n directions, batch size, hid dim]

        #n layers and n directions in the decoder will both always be 1, therefore:
        #hidden = [1, batch size, hid dim]
        #context = [1, batch size, hid dim]

        input = input.unsqueeze(0) # irá criar um vetor [...]

        #input = [1, batch size]

        embedded = self.dropout(self.embedding(input))

        #embedded = [1, batch size, emb dim]

        emb_con = torch.cat((embedded, context), dim=2)

        #emb_con = [1, batch size, emb dim + hid dim]

        output, hidden = self.rnn(emb_con, hidden)

        #output = [sent len, batch size, hid dim * n directions]
        #hidden = [n layers * n directions, batch size, hid dim]

        #sent len, n layers and n directions will always be 1 in the decoder, therefore:
        #output = [1, batch size, hid dim]
        #hidden = [1, batch size, hid dim]

        output = torch.cat((embedded.squeeze(0), hidden.squeeze(0), context.squeeze(0)), dim=1)

        #output = [batch size, emb dim + hid dim * 2]

        prediction = self.out(output)

        #prediction = [batch size, output dim]

        return prediction, hidden
```

### Seq2Seq

Agora, vamos juntar todas as peças de nosso quebra-cabeças para gerar um modelo de sequência-para-sequência (`seq2seq`).

```python
class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder, device):
        super().__init__()

        self.encoder = encoder
        self.decoder = decoder
        self.device = device

        assert encoder.hid_dim == decoder.hid_dim, \
            "Hidden dimensions of encoder and decoder must be equal!"

    def forward(self, src, trg, teacher_forcing_ratio=0.5):

        #src = [src sent len, batch size]
        #trg = [trg sent len, batch size]
        #teacher_forcing_ratio is probability to use teacher forcing
        #e.g. if teacher_forcing_ratio is 0.75 we use ground-truth inputs 75% of the time

        batch_size = trg.shape[1]
        max_len = trg.shape[0]
        trg_vocab_size = self.decoder.output_dim

        #tensor to store decoder outputs
        outputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device)

        #last hidden state of the encoder is the context
        context = self.encoder(src)

        #context also used as the initial hidden state of the decoder
        hidden = context

        #first input to the decoder is the <sos> tokens
        input = trg[0,:]

        for t in range(1, max_len):

            output, hidden = self.decoder(input, hidden, context)
            outputs[t] = output
            teacher_force = random.random() < teacher_forcing_ratio
            top1 = output.max(1)[1]
            input = (trg[t] if teacher_force else top1)

        return outputs
```

```python

```

```python
INPUT_DIM = len(SRC.vocab)
OUTPUT_DIM = len(TRG.vocab)
ENC_EMB_DIM = 256
DEC_EMB_DIM = 256
HID_DIM = 512
ENC_DROPOUT = 0.5
DEC_DROPOUT = 0.5

enc = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, ENC_DROPOUT)
dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, HID_DIM, DEC_DROPOUT)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = Seq2Seq(enc, dec, device).to(device)
```

```python

```

```python
def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')
```

    The model has 14,220,293 trainable parameters

```python

```

```python
optimizer = optim.Adam(model.parameters())
```

```python

```

```python
pad_idx = TRG.vocab.stoi['<pad>']

criterion = nn.CrossEntropyLoss(ignore_index=pad_idx)
```

```python

```

```python
def train(model, iterator, optimizer, criterion, clip):

    model.train()

    epoch_loss = 0

    for i, batch in enumerate(iterator):

        src = batch.src
        trg = batch.trg

        optimizer.zero_grad()

        output = model(src, trg)

        #trg = [trg sent len, batch size]
        #output = [trg sent len, batch size, output dim]

        output = output[1:].view(-1, output.shape[-1])
        trg = trg[1:].view(-1)

        #trg = [(trg sent len - 1) * batch size]
        #output = [(trg sent len - 1) * batch size, output dim]

        loss = criterion(output, trg)

        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)

        optimizer.step()

        epoch_loss += loss.item()

    return epoch_loss / len(iterator)
```

```python

```

```python
def evaluate(model, iterator, criterion):

    model.eval()

    epoch_loss = 0

    with torch.no_grad():

        for i, batch in enumerate(iterator):

            src = batch.src
            trg = batch.trg

            output = model(src, trg, 0) #turn off teacher forcing

            #trg = [trg sent len, batch size]
            #output = [trg sent len, batch size, output dim]

            output = output[1:].view(-1, output.shape[-1])
            trg = trg[1:].view(-1)

            #trg = [(trg sent len - 1) * batch size]
            #output = [(trg sent len - 1) * batch size, output dim]

            loss = criterion(output, trg)

            epoch_loss += loss.item()

    return epoch_loss / len(iterator)
```

```python

```

```python
def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs
```

Training this model without a CUDA takes 6h+ no meu velho Macbook Pro 2013

```python
N_EPOCHS = 10
CLIP = 1
SAVE_DIR = 'models'
MODEL_SAVE_PATH = os.path.join(SAVE_DIR, 'tut2_model.pt')

best_valid_loss = float('inf')

if not os.path.isdir(f'{SAVE_DIR}'):
    os.makedirs(f'{SAVE_DIR}')

for epoch in range(N_EPOCHS):

    start_time = time.time()

    train_loss = train(model, train_iterator, optimizer, criterion, CLIP)
    valid_loss = evaluate(model, valid_iterator, criterion)

    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)

    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), MODEL_SAVE_PATH)

    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')
```

    Epoch: 01 | Time: 13m 14s
    	Train Loss: 4.641 | Train PPL: 103.694
    	 Val. Loss: 4.319 |  Val. PPL:  75.093

```python

```

```python
model.load_state_dict(torch.load(MODEL_SAVE_PATH))

test_loss = evaluate(model, test_iterator, criterion)

print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')
```

```python

```

```python

```

```python

```

```python

```

```python

```


# Selecting intracellular ions and creating a new METASPACE annotation database
This notebook shows how to select intracellular metabolites from processed SpaceM data.
Ion-cell colocalization is quantified using either Spearman correlation or an area-**n**ormalized **M**anders **C**olocalization **C**oefficient (NMCC).
Selected ions can be used to create a custom annotation database for use on METASPACE.

You can also download this notebook [here](https://git.embl.de/mattausc/outer-spacem/-/raw/master/docs/examples/intracell_selection/create_database.ipynb?inline=false) or as part of our [collection of SpaceM analysis notebooks](https://git.embl.de/grp-alexandrov/spacem-analysis-notebooks).

Before running this notebook, make sure to...

- [Install the METASPACE Python client](https://pypi.org/project/metaspace2020/)
- [Create an API key for METASPACE](https://metaspace2020.eu/user/me)
- [Register your API key locally](https://metaspace2020.readthedocs.io/en/latest/content/examples/fetch-dataset-annotations.html#Connect-to-the-sm-server)

```python
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import re
from pathlib import Path

from scipy.stats import spearmanr

from metaspace import SMInstance

%matplotlib inline
%config InlineBackend.figure_formats = ['retina']
```

```python
sns.set(
    rc={
        "figure.figsize":(7.5,7.5),
        "legend.frameon": False
    },
    style="ticks",
    context="notebook"
)
```

### Setup

Quantification of ion-cell localization is done on the spatiomolecular matrices of **ablation marks** that are produced (i.e. the matrices stored in the `analysis/ablation_mark_analysis` directory, from here on called **AM matrix**).
For this example we're going to use the AM matrices of three SpaceM datasets.

From experience it makes close to no difference for ion selection whether ion-cell colocalization is evaluated on the combined datasets together, or separately and then aggregated. This notebook computes the scores separately and then aggregates them, so there is no need to concatenate all your dataseset before running them through this notebook.

**Note:** For this notebook you're going to need the AM matrices' .csv files, not the .h5ad ones!

```python
# Datasets to be used to select intracellular ions

dataset_paths = [
    # "/Users/alberto-mac/Documents/DA_ESPORTARE/LOCAL_EMBL_FILES/scratch/projects/gastrosome_processing/SpaceM_processing_new/Drug_W8/analysis/ablation_mark_analysis/spatiomolecular_matrix.csv",
    "/Users/alberto-mac/Documents/DA_ESPORTARE/LOCAL_EMBL_FILES/scratch/projects/gastrosome_processing/SpaceM_processing_new/Feeding_W3/analysis/ablation_mark_analysis/spatiomolecular_matrix.csv"
]
```

#### Which selection method should I use?
- Use `'nmcc'` if you only need high precision for cells (i.e. ions should be exclusive to cells but do not need to be present in all cells in a dataset)
- Use `'spearman'` if you want your ions to have high precision *and* recall for cells (i.e. ions should be exclusive to cells and also present in all cells)

An ion's NMCC is defined as
\begin{equation}
\frac{\text{on-sample ion counts}}{\text{on-sample ion counts + off-sample counts}} \cdot \frac{\text{off-sample area}}{\text{on-sample area}}
\end{equation}
Or, more precisely:
\begin{equation}
M = \frac{\sum_i I_i \cdot f_i}{\sum_i I_i \cdot f_i + \sum_i I_i \cdot (1-f_i)\cdot \frac{\overline{f}}{1 - \overline{f}}}
\end{equation}

Where $I_i$ is the ion's intensity and $f_i$ is ablation mark sampling ratio for pixel $i$.
Essentially it describes the fraction of ions that are on-sample when off- and on-sample areas are the same size.
An NMCC of 0 therefore indicates that all ions are off-sample (i.e. exclusively extracellular), 0.5 indicates that ions are evenly distributed between on- and off-sample area, and 1 indicates that all ions are exclusively on-sample (i.e. intracellular).

#### Filtering out low-abundance ions
Low occurence ions have shown to have very little contribution to overall variance in SpaceM datasets and have close to zero impact on clusterings.
Unless you specifically want to analyse these low-occurence ions, you can exclude them here from your database using the `nonzero_thresold` parameter. By default, only ions that are present in >2.5% of pixels in at least one dataset are included.
If you don't want to exclude low-abundance ions, set this parameter to 0.

```python
selection_method = "nmcc" # "nmcc" or "spearman"
selection_threshold = 0.6
# Recommended for spearman: 0.1-0.2
# Recommended for nmcc: 0.6-0.7

nonzero_threshold = 0. # 0.025 # Only include ions present in > 2.5% of pixels in >= 1 dataset(s). Set to 0 to disable.
```

### Evaluate & select intracellular ions

```python
def nmcc(I, f):
    f_ = f.mean()
    I_on = np.sum(I * f)
    I_off = np.sum(I * (1 - f) * (f_ / (1 - f_)))
    return I_on / (I_on + I_off)
```

##### Load the datasets

```python
am_matrices = []

ion_regex = r"^C[0-9]*.*?[\+-]" # This excludes SpaceM's image analysis features. You might want to adjust this if you want to include non-carbon metabolites!

for ds_idx, ds_path in enumerate(dataset_paths):
    print(ds_idx, ds_path)
    amm = pd.read_csv(ds_path)
    amm["dataset"] = ds_idx
    am_matrices.append(amm)

am_matrix = pd.concat(am_matrices, ignore_index=True).fillna(0)
ions = am_matrix.columns.to_series().filter(regex=ion_regex).values

print(f"Read {len(am_matrices)} datasets containing {len(ions)} unique ion annotations for {len(am_matrix)} pixels.")
```

    0 /Users/alberto-mac/Documents/DA_ESPORTARE/LOCAL_EMBL_FILES/scratch/projects/gastrosome_processing/SpaceM_processing_new/Feeding_W3/analysis/ablation_mark_analysis/spatiomolecular_matrix.csv
    Read 1 datasets containing 820 unique ion annotations for 22500 pixels.

```python
scores = pd.DataFrame()

for dataset, ds_amm in am_matrix.groupby(["dataset"]):
    for ion in ions:
        if (ds_amm[ion] != 0).mean() > nonzero_threshold:
            ion_scores = {
                "dataset": dataset,
                "ion": ion,
                "spearman": spearmanr(ds_amm[ion], ds_amm["am_sampling_ratio"]).correlation,
                "nmcc": nmcc(ds_amm[ion], ds_amm["am_sampling_ratio"])
            }
            scores = pd.concat([scores, pd.DataFrame(ion_scores, index=[0])], ignore_index=True)

scores_agg = scores.groupby(["ion"])[["spearman", "nmcc"]].mean()
scores_agg["selected"] = scores_agg[selection_method] > selection_threshold
```

Once we have computed the scores, we can take a closer look at the distribution of per-ion scores for both spearman correlation and NMCC:

```python
g = sns.displot(
    scores_agg.melt(value_vars = ["spearman", "nmcc"]),
    x="value",
    col="variable",
    kind="hist",
    bins=50,
    fill=True,
    linewidth=0,
    alpha=1
)

g.set_titles("")

for ax, title in zip(g.axes.flatten(), g.col_names):
    ax.axvline(x=selection_threshold, c="0.2", linestyle=":")
    ax.text(
        s="Threshold",
        x=selection_threshold+0.05,
        y=plt.ylim()[1],
        fontsize=10,
        va="top",
        rotation=90
    )
    if title == selection_method:
        ax.set_xlabel(title + "\n(selected method)")
    else:
        ax.set_xlabel(title)

plt.xlim(-1, 1)
ax.locator_params(nbins=4)

plt.show()
```

Ions for which the selected score is above the selection threshold will be included in your custom database.

```python
sns.scatterplot(
    data=scores_agg,
    x="nmcc",
    y="spearman",
    linewidth=0,
    hue="selected",
    s=25
)

plt.axhline(y=0, c=".2", linestyle=":", zorder=0)
plt.axvline(x=0.5, c="0.2", linestyle=":", zorder=0)

plt.xlim([0, 1])
plt.ylim([-1, 1])

plt.locator_params(nbins=4)
sns.despine()
plt.show()
```

Note that the two different scores are are not proportional, therefore depending on the chosen method you may select different ions.

- Spearman correlation will favor ions that are intracellular but also present in the majority of cells.
- NMCC will favor all ions that are intracellular, but not necessarily present in all cells.

```python
selected_ions = scores_agg[
    scores_agg["selected"]
].index

selected_metabolites = np.unique([re.match(r"^(.+?)[\+-]", x).group(1) for x in selected_ions])

print(f"Selected {selected_metabolites.size} metabolites.")
```

    Selected 171 metabolites.

### Create & upload new database

An easy way to now restrict your single-cell analysis to intrallular ions is to exclude all ions that are not part of `selected_metabolites`. An even better way however is to use this list of ions and turn it into a custom database for METASPACE and reannotate your datasets, as using a custom database will also combat the issue of missing values due to METASPACE's FDR threshold.

To create a custom database, you're going to need a name and a version number:

```python
db_name = "Gastrosome_FeedingW3_intra_ions"
db_version = "v1"
# db_name = "Gastrosome_DrugW8_intra_ions"
# db_version = "v2"

db_filename = f"AB_{db_name}_{db_version}.tsv" # XY = ideally your initials for easier handling on METASPACE

import os
out_file = os.path.join("/Users/alberto-mac/EMBL_ATeam/projects/gastrosome", db_filename)
```

```python
df_df = pd.DataFrame(
    {
        "name": selected_metabolites,
        "formula": selected_metabolites
    },
    index = pd.RangeIndex(start=1, stop=selected_metabolites.size + 1, name="id")
)
df_df.to_csv(
    db_filename, sep="\t", index=True
)
df_df.to_csv(
    out_file, sep="\t", index=True
)
```

Once we have the database file, we can upload it to METASPACE:

```python
sm = SMInstance()
```

```python
# This will prompt you to enter your API key if needed and it will save it to a config file.
# Note that API keys should be kept secret like passwords.
# sm.save_login()
```

```python
upload_result = sm.create_database(
    local_path=db_filename,
    name=db_name,
    version=db_version,
    is_public=False,
)

upload_result
```

    Uploading part   1/  1 of AB_Gastrosome_FeedingW3_intra_ions_v1.tsv file...

    {'id': 471}

This chunk of code will create the database and return a number corresponding to your database's ID. Write it down and continue with the [reannotation notebook](bulk_reannotation.ipynb) to reannotate your datasets against the new database.


# Notebook 3: Evaluating prediction quality with __Solaris__

Here, we're going to try running several different pre-trained models to generate predictions on the exact same image to help us decide which one works best.

## Sections

1. [__Running inference with the other pre-trained models__](#Running-inference-with-the-other-pre-trained-models)
2. [__Pixel-wise metrics for prediction quality__](#Pixel-wise-metrics-for-prediction-quality)
3. [__Object-wise metrics for prediction quality__](#Object-wise-metrics-for-prediction-quality)

While we talk through evaluation metrics, it'll be helpful to have a few different models' outputs to compare. To this end, we're going to run inference with all four of the pre-trained models available in `solaris` on the sample image we've used previously. The following cell runs that inference pipeline for the other three models, just like we did for XD_XD's model earlier; let's get it started!

## Running inference with the other pre-trained models

```python
import solaris as sol
import numpy as np
import skimage
import os
from matplotlib import pyplot as plt
import time

data_path = '/data'  # NON-WORKSHOP PARTICIPANTS: change this path to point to the directory where you've stored the data.

def run_inference(config_path):
    print('Loading config...')
    config = sol.utils.config.parse(config_path)
    print('Config loaded. Initializing model...')
    inferer = sol.nets.infer.Inferer(config)
    print('Model initialized. Loading dataset...')
    inf_df = sol.nets.infer.get_infer_df(config)
    print('Dataset loaded. Running inference on the image...this could take a bit, be patient.')
    start_time = time.time()
    inferer(inf_df)
    end_time = time.time()
    print('Running inference on one image took {} seconds.'.format(end_time-start_time))
    print('Vectorizing output...')
    resulting_preds = skimage.io.imread(os.path.join(config['inference']['output_dir'], 'MVOI_nadir10_test_sample.tif'))
    # handle issue we currently have with multi-channel masks and automated footprint generation
    if config['data_specs']['mask_channels'] > 1:
        resulting_preds = resulting_preds[:, :, 0]
    predicted_footprints = sol.vector.mask.mask_to_poly_geojson(
        pred_arr=resulting_preds,
        reference_im=os.path.join(data_path, 'MVOI_data/MVOI_nadir10_test_sample.tif'),
        do_transform=True)
    print('Output vectorized.')
    predicted_footprints.to_file(os.path.join(config['inference']['output_dir'], 'MVOI_nadir10_predictions.geojson'), driver='GeoJSON')
    print('Done.')
```

```python
for config_path in ('workshop_configs/selimsef_densenet121unet_workshop_infer.yml',
                    'workshop_configs/selimsef_densenet161unet_workshop_infer.yml',
                    'workshop_configs/selimsef_resnet34unet_workshop_infer.yml',
                    'workshop_configs/xdxd_workshop_infer.yml'):
    run_inference(os.path.join(data_path, config_path))
```

When the above cell finishes, we'll visualize the outputs.

#### Visualizing the outputs

Though it's preferable to work with vector-formatted labels, the raster versions are generally easier to visualize with `matplotlib`, so we'll show those here for comparison purposes.

```python
output_dirs = ['xdxd_inference_out', 'selimsef_resnet34_inference_out',
               'selimsef_densenet121_inference_out', 'selimsef_densenet161_inference_out']
model_names = ['VGG16', 'ResNet34', 'DenseNet121', 'DenseNet161']

f, axarr = plt.subplots(2, 3, figsize=(16, 12))

for i in range(4):
    im = skimage.io.imread(os.path.join(output_dirs[i], 'MVOI_nadir10_test_sample.tif'))
    im = (im > 0).astype('uint8')[:, :, 0]
    axarr[i//2, i%2+1].imshow(im, cmap='gray')
    axarr[i//2, i%2+1].set_title(model_names[i] + ' predictions', size=16)
    axarr[i//2, i%2+1].axis('off')

ground_truth = sol.vector.mask.footprint_mask(
    os.path.join(data_path, 'MVOI_data/MVOI_nadir10_test_sample.geojson'),
    reference_im = os.path.join(data_path, 'MVOI_data/MVOI_nadir10_test_sample.tif'))

axarr[0, 0].imshow(ground_truth, cmap='gray')
axarr[0, 0].set_title('Ground Truth labels', size=16)
axarr[0, 0].axis('off')

# switch B and R for viz
im_arr = skimage.io.imread(os.path.join(data_path, 'MVOI_data/viz_version.tif'))
axarr[1, 0].imshow(im_arr[:, :, 0:3])
axarr[1, 0].set_title('Source image', size=16)
axarr[1, 0].axis('off');

```

As you can see, the models are _generally_ very similar in their predictions, but there are some subtle differences. Some of these may have dramatic effects on how their performance is scored! But how would we go about deciding which is best?

__Pause here for slides: CosmiQ_Solaris_Evaluating_Performance__

## Pixel-wise metrics for prediction quality

We'll start by evaluating model performance with a couple of pixel-wise metrics, i.e., metrics that score models based on what fraction of the pixels are correct.

Let's start with "accuracy", since that's the most common term used when discussing model performance. Few people are actually familiar wtih the definition of accuracy:

$
\begin{align}
accuracy=\frac{TN+TP}{TN+TP+FN+FP}
\end{align}
$

Where:
- TN = True Negative (a pixel is ID'ed as non-building, and is in fact not a building)
- TP = True Positive (a pixel is ID'ed as building, and is in fact a building)
- FN = False Negative (a pixel is ID'ed as non-building, when it is, in fact, building)
- FP = False Positive (a pixel is ID'ed as building, but it's actually background)

In layperson's terms, this metric is essentially what fraction of pixels are correct. Let's calculate that for each model above.

```python
def acc(prediction, ground_truth):
    return np.sum(prediction == ground_truth)/float(prediction.size)

output_dirs = ['xdxd_inference_out', 'selimsef_resnet34_inference_out',
               'selimsef_densenet121_inference_out', 'selimsef_densenet161_inference_out']
model_names = ['VGG16\nU-Net', 'ResNet34\nU-Net', 'DenseNet121\nU-Net', 'DenseNet161\nU-Net']

ground_truth = ground_truth > 0

accuracy_scores = []

for i in range(4):
    pred_image = skimage.io.imread(os.path.join(output_dirs[i], 'MVOI_nadir10_test_sample.tif'))
    pred_image = pred_image > 0
    pred_image = pred_image[:, :, 0]
    accuracy = acc(pred_image, ground_truth)
    print("{} accuracy: {}".format(model_names[i], accuracy))
    accuracy_scores.append(accuracy)

plt.bar(model_names, accuracy_scores, color='black')
plt.ylabel('Accuracy', size=16)
plt.ylim((0.85, 1))
```

These scores are _very_ similar - tiny fractions of a percent different - and more or less impossible to detect, even zoomed way in on the graph (note the Y axis on the graph above). What if we try a different metric, like $F_1$ score:

$
\begin{align}
F_1 = 2\times\frac{P\times R}{P+R}
\end{align}
$

Where
- P = Precision:

$
\begin{align}
\frac{TP}{TP+FP}
\end{align}
$
- R = Recall:

$
\begin{align}
\frac{TP}{TP+FN}
\end{align}
$

```python
f1_scores = []
precision = []
recall = []
for i in range(4):
    pred_image = skimage.io.imread(os.path.join(output_dirs[i], 'MVOI_nadir10_test_sample.tif'))
    pred_image = pred_image > 0
    pred_image = pred_image[:, :, 0]
    f1, prec, rec = sol.eval.pixel.f1(ground_truth, pred_image)
#    print("{} F1: {}".format(model_names[i], results[0]))
#    print("{} Precision: {}".format(model_names[i], results[1]))
#    print("{} Recall: {}".format(model_names[i], results[2]))
    f1_scores.append(f1)
    precision.append(prec)
    recall.append(rec)

f, axarr = plt.subplots(1, 3, figsize=(10, 4))
f.subplots_adjust(wspace=0.6)
axarr[0].bar(model_names, f1_scores)
axarr[0].set_xticklabels(labels=model_names, rotation=90)
axarr[0].set_ylabel('$F_1$ Score', size=16)
axarr[1].bar(model_names, precision)
axarr[1].set_xticklabels(labels=model_names, rotation=90)
axarr[1].set_ylabel('Precision', size=16)
axarr[2].bar(model_names, recall)
axarr[2].set_xticklabels(labels=model_names, rotation=90)
axarr[2].set_ylabel('Recall', size=16);
```

This shows _some_ pixel-wise difference between these different predictions - specifically, the VGG16 model provides _slightly_ better prediction quality on a pixel-by-pixel basis. The Precision and Recall plots shed a bit more light.

This still leaves potential for problems, though. The two examples below produce the exact same $F_1$ score, Precision, and Recall on a pixel-wise basis:

Depending on your use case, you may want to be able to distinguish between these two sets of predictions and optimize for one rather than the other. To do so, we'll need something more than pixel-wise metrics.

## Object-wise metrics for prediction quality
At CosmiQ Works, we generally use object-wise metrics to evaluate model performance for building footprint extraction tasks (or other object identification tasks).
This involves a two-step process, adapted from the [ImageNet](www.image-net.org) competitions:

1. Call positive and negative building identifications using an Intersection-over-Union (IoU) threshold
2. Calculate the $F_1$ score of identifications at an object level (rather than a pixel level)

The graphic below displays how we calculate IoU for a prediction.

`solaris` provides functionality to do this calculation for all of the objects within a given set of vector labels, providing both the object-by-object IoU scores.

Next, we set a threshold for how high a prediction's IoU must be with a ground truth object to be called "successful" (i.e. a True Positive). We generally set this threshold at __0.5__ for the SpaceNet Challenges. This allows us to identify specific predictions as True Positives and False Positives, as well as identify False Negatives where there wasn't a high-quality prediction. We call this metric the __SpaceNet Metric__.

Using that threshold, let's assess the SpaceNet Metric for the four prediction sets we generated earlier:

```python
evaluator = sol.eval.base.Evaluator(os.path.join(data_path, 'MVOI_data/MVOI_nadir10_test_sample.geojson'))
prediction_dirs = ['xdxd_inference_out', 'selimsef_resnet34_inference_out',
               'selimsef_densenet121_inference_out', 'selimsef_densenet161_inference_out']
model_names = ['VGG16\nU-Net', 'ResNet34\nU-Net', 'DenseNet121\nU-Net', 'DenseNet161\nU-Net']

f1_scores = []
precision = []
recall = []
for i in range(4):
    evaluator.load_proposal(os.path.join(prediction_dirs[i],'MVOI_nadir10_predictions.geojson'),
                            pred_row_geo_value='geometry',
                            conf_field_list=[])
    results = evaluator.eval_iou(miniou=0.5, calculate_class_scores=False)
    f1_scores.append(results[0]['F1Score'])
    precision.append(results[0]['Precision'])
    recall.append(results[0]['Recall'])

f, axarr = plt.subplots(1, 3, figsize=(10, 4))
f.subplots_adjust(wspace=0.6)
axarr[0].bar(model_names, f1_scores)
axarr[0].set_xticklabels(labels=model_names, rotation=90)
axarr[0].set_ylabel('$F_1$ Score', size=16)
axarr[1].bar(model_names, precision)
axarr[1].set_xticklabels(labels=model_names, rotation=90)
axarr[1].set_ylabel('Precision', size=16)
axarr[2].bar(model_names, recall)
axarr[2].set_xticklabels(labels=model_names, rotation=90)
axarr[2].set_ylabel('Recall', size=16);

```

These are still close. What if we increase the IoU threshold to 0.85, meaning predictions must be much more similar to the ground truth to be deemed "correct"?

```python
f1_scores = []
precision = []
recall = []
for i in range(4):
    evaluator.load_proposal(os.path.join(prediction_dirs[i],'MVOI_nadir10_predictions.geojson'),
                            pred_row_geo_value='geometry',
                            conf_field_list=[])
    results = evaluator.eval_iou(miniou=0.75, calculate_class_scores=False)
    f1_scores.append(results[0]['F1Score'])
    precision.append(results[0]['Precision'])
    recall.append(results[0]['Recall'])

f, axarr = plt.subplots(1, 3, figsize=(10, 4))
f.subplots_adjust(wspace=0.6)
axarr[0].bar(model_names, f1_scores)
axarr[0].set_xticklabels(labels=model_names, rotation=90)
axarr[0].set_ylabel('$F_1$ Score', size=16)
axarr[1].bar(model_names, precision)
axarr[1].set_xticklabels(labels=model_names, rotation=90)
axarr[1].set_ylabel('Precision', size=16)
axarr[2].bar(model_names, recall)
axarr[2].set_xticklabels(labels=model_names, rotation=90)
axarr[2].set_ylabel('Recall', size=16)
```

It's notable that 1. all of the scores drop substantially, but 2. the VGG16 model now achieves ~18% better scores than the other models at the higher IoU threshold! These are truly state-of-the-art models for overhead imagery analysis, and the relatively low scores highlight that there's still a lot of room for improvement in this field. We hope some of you will use these tools to make this happen!

__Coming up next:__ How well do pre-trained models work on images of geographies that they've never seen before, and fine-tuning models to improve that performance.