```python
!pip install qiskit
!pip install qiskit-machine-learning
```

    Collecting qiskit
      Downloading qiskit-0.33.0.tar.gz (13 kB)
    Collecting qiskit-terra==0.19.0
      Downloading qiskit_terra-0.19.0-cp37-cp37m-manylinux2010_x86_64.whl (6.4 MB)
         |████████████████████████████████| 6.4 MB 7.2 MB/s
    Collecting qiskit-aer==0.9.1
      Downloading qiskit_aer-0.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (17.9 MB)
         |████████████████████████████████| 17.9 MB 344 kB/s
    Collecting qiskit-ibmq-provider==0.18.1
      Downloading qiskit_ibmq_provider-0.18.1-py3-none-any.whl (237 kB)
         |████████████████████████████████| 237 kB 40.4 MB/s
    Collecting qiskit-ignis==0.7.0
      Downloading qiskit_ignis-0.7.0-py3-none-any.whl (200 kB)
         |████████████████████████████████| 200 kB 51.8 MB/s
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.9.1->qiskit) (1.19.5)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.9.1->qiskit) (1.4.1)
    Collecting requests-ntlm>=1.1.0
      Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Collecting websocket-client>=1.0.1
      Downloading websocket_client-1.2.3-py3-none-any.whl (53 kB)
         |████████████████████████████████| 53 kB 1.0 MB/s
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.1->qiskit) (2.8.2)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.1->qiskit) (1.24.3)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.1->qiskit) (2.23.0)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.10.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)
         |████████████████████████████████| 1.4 MB 45.7 MB/s
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.0->qiskit) (0.3.4)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.0->qiskit) (1.7.1)
    Collecting stevedore>=3.0.0
      Downloading stevedore-3.5.0-py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.0 MB/s
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.0->qiskit) (5.4.8)
    Collecting ply>=3.10
      Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 4.2 MB/s
    Collecting symengine>=0.8
      Downloading symengine-0.8.1-cp37-cp37m-manylinux2010_x86_64.whl (38.2 MB)
         |████████████████████████████████| 38.2 MB 1.2 MB/s
    Collecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (943 kB)
         |████████████████████████████████| 943 kB 34.8 MB/s
    Collecting scipy>=1.0
      Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
         |████████████████████████████████| 38.1 MB 1.2 MB/s
    Collecting python-constraint>=1.4
      Downloading python-constraint-1.4.0.tar.bz2 (18 kB)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.1->qiskit) (1.15.0)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.1->qiskit) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.1->qiskit) (2021.10.8)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.1->qiskit) (3.0.4)
    Collecting ntlm-auth>=1.0.2
      Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Collecting cryptography>=1.3
      Downloading cryptography-36.0.0-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
         |████████████████████████████████| 3.6 MB 44.1 MB/s
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.1->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.1->qiskit) (2.21)
    Collecting pbr!=2.1.0,>=2.0.0
      Downloading pbr-5.8.0-py2.py3-none-any.whl (112 kB)
         |████████████████████████████████| 112 kB 74.8 MB/s
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.0->qiskit) (4.8.2)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.0->qiskit) (3.6.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.0->qiskit) (3.10.0.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.19.0->qiskit) (1.2.1)
    Building wheels for collected packages: qiskit, python-constraint
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.33.0-py3-none-any.whl size=11763 sha256=9c6b76f5e5ce7ce8f189d285044d28477e6af1a14502b1a93d437f5f670e2263
      Stored in directory: /root/.cache/pip/wheels/d4/60/ef/c49ae113df02818c4ade5fb6e2e89bb928625412eda0bb25f5
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24081 sha256=5718fdc37d2db9183104b1d851bd3a0eeab95114197d7ba73f2b5b52102d08bb
      Stored in directory: /root/.cache/pip/wheels/07/27/db/1222c80eb1e431f3d2199c12569cb1cac60f562a451fe30479
    Successfully built qiskit python-constraint
    Installing collected packages: pbr, tweedledum, symengine, stevedore, scipy, retworkx, python-constraint, ply, ntlm-auth, cryptography, websocket-client, requests-ntlm, qiskit-terra, qiskit-ignis, qiskit-ibmq-provider, qiskit-aer, qiskit
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed cryptography-36.0.0 ntlm-auth-1.5.0 pbr-5.8.0 ply-3.11 python-constraint-1.4.0 qiskit-0.33.0 qiskit-aer-0.9.1 qiskit-ibmq-provider-0.18.1 qiskit-ignis-0.7.0 qiskit-terra-0.19.0 requests-ntlm-1.1.0 retworkx-0.10.2 scipy-1.7.3 stevedore-3.5.0 symengine-0.8.1 tweedledum-1.1.1 websocket-client-1.2.3
    Collecting qiskit-machine-learning
      Downloading qiskit_machine_learning-0.2.1-py3-none-any.whl (96 kB)
         |████████████████████████████████| 96 kB 3.8 MB/s
    Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from qiskit-machine-learning) (1.19.5)
    Requirement already satisfied: qiskit-terra>=0.18.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-machine-learning) (0.19.0)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-machine-learning) (5.4.8)
    Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-machine-learning) (1.0.1)
    Requirement already satisfied: fastdtw in /usr/local/lib/python3.7/dist-packages (from qiskit-machine-learning) (0.3.4)
    Requirement already satisfied: scipy>=1.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-machine-learning) (1.7.3)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-machine-learning) (57.4.0)
    Requirement already satisfied: retworkx>=0.10.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (0.10.2)
    Requirement already satisfied: symengine>=0.8 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (0.8.1)
    Requirement already satisfied: python-constraint>=1.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (1.4.0)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (2.8.2)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (0.3.4)
    Requirement already satisfied: tweedledum<2.0,>=1.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (1.1.1)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (1.7.1)
    Requirement already satisfied: ply>=3.10 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (3.11)
    Requirement already satisfied: stevedore>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra>=0.18.0->qiskit-machine-learning) (3.5.0)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-terra>=0.18.0->qiskit-machine-learning) (1.15.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.20.0->qiskit-machine-learning) (1.1.0)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.20.0->qiskit-machine-learning) (3.0.0)
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra>=0.18.0->qiskit-machine-learning) (4.8.2)
    Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra>=0.18.0->qiskit-machine-learning) (5.8.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra>=0.18.0->qiskit-machine-learning) (3.6.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra>=0.18.0->qiskit-machine-learning) (3.10.0.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra>=0.18.0->qiskit-machine-learning) (1.2.1)
    Installing collected packages: qiskit-machine-learning
    Successfully installed qiskit-machine-learning-0.2.1

```python
import numpy as np

# Importing standard Qiskit libraries
import qiskit
from qiskit import QuantumCircuit, transpile, Aer, IBMQ
from qiskit.tools.jupyter import *
from qiskit.visualization import *
#from ibm_quantum_widgets import *
from qiskit.providers.aer import QasmSimulator

# Loading your IBM Quantum account(s)
#provider = IBMQ.load_account()
```

```python
import numpy as np
import matplotlib.pyplot as plt

from qiskit import Aer, QuantumCircuit
from qiskit.opflow import Z, I, StateFn
from qiskit.utils import QuantumInstance
from qiskit.circuit import Parameter
from qiskit.circuit.library import RealAmplitudes, ZZFeatureMap
from qiskit.algorithms.optimizers import COBYLA, L_BFGS_B

from qiskit_machine_learning.neural_networks import TwoLayerQNN, CircuitQNN
from qiskit_machine_learning.algorithms.classifiers import NeuralNetworkClassifier, VQC
from qiskit_machine_learning.algorithms.regressors import NeuralNetworkRegressor, VQR

from typing import Union

from qiskit_machine_learning.exceptions import QiskitMachineLearningError

from IPython.display import clear_output
```

```python
  #def get_noise_model():
from qiskit.providers.aer.noise import NoiseModel
from qiskit.providers.aer import noise
noise_model = NoiseModel()
# Error probabilities
#prob_1 = 0.001  # 1-qubit gate
prob_2 = 0.000   # 2-qubit gate

# Depolarizing quantum errors
#error_1 = noise.depolarizing_error(prob_1, 1)
error_2 = noise.depolarizing_error(prob_2, 2)

# Add errors to noise model
noise_model = noise.NoiseModel()
#noise_model.add_all_qubit_quantum_error(error_1, ['u1', 'u2', 'u3', 'h', 'id', 'rz', 'sx', 'rx', 'ry'])
noise_model.add_all_qubit_quantum_error(error_2, ['cx'])

    #return noise_model
```

```python
quantum_instance = QuantumInstance(Aer.get_backend('aer_simulator'), shots=1024)
```

```python
#noise_model1 = get_noise_model()
quantum_instance_noise = QuantumInstance(Aer.get_backend('aer_simulator'), shots=1024, noise_model = noise_model)

```

```python
# construct simple feature map
param_x = Parameter('x')
feature_map = QuantumCircuit(1, name='fm')
feature_map.ry(param_x, 0)
print(feature_map)

# construct simple ansatz
param_y = Parameter('y')
ansatz = QuantumCircuit(1, name='vf')
ansatz.ry(param_y, 0)
print(ansatz)
#ansatz.draw(output='mpl')
# construct QNN
regression_opflow_qnn = TwoLayerQNN(1, feature_map, ansatz, quantum_instance=quantum_instance)
regression_opflow_qnn_noise = TwoLayerQNN(1, feature_map, ansatz, quantum_instance=quantum_instance_noise)
```

       ┌───────┐
    q: ┤ Ry(x) ├
       └───────┘
       ┌───────┐
    q: ┤ Ry(y) ├
       └───────┘

```python
# callback function that draws a live plot when the .fit() method is called
objective_func_vals = []
def callback_graph(weights, obj_func_eval):
    clear_output(wait=True)
    objective_func_vals.append(obj_func_eval)
    plt.title("Objective function value against iteration")
    plt.xlabel("Iteration")
    plt.ylabel("Objective function value")
    plt.plot(range(len(objective_func_vals)), objective_func_vals)
    plt.show()
```

```python
# callback function that draws a live plot when the .fit() method is called
objective_func_vals_noise = []
def callback_graph_noise(weights, obj_func_eval):
    clear_output(wait=True)
    objective_func_vals_noise.append(obj_func_eval)
    plt.title("Objective function value against iteration")
    plt.xlabel("Iteration")
    plt.ylabel("Objective function value")
    plt.plot(range(len(objective_func_vals_noise)), objective_func_vals_noise)
    plt.show()
```

```python
# construct the regressor from the neural network
regressor = NeuralNetworkRegressor(neural_network=regression_opflow_qnn,
                                   loss='l2',
                                   optimizer=L_BFGS_B(),
                                   callback=callback_graph)
```

```python
# construct the regressor from the neural network
regressor_noise = NeuralNetworkRegressor(neural_network=regression_opflow_qnn_noise,
                                   loss='l2',
                                   optimizer=L_BFGS_B(),
                                   callback=callback_graph_noise)
```

```python
#from IPython.core.debugger import set_trace
```

```python
#import sys
'''
sys.path.append("../Qiskit_Dynamic_Modelling")
sys.path.append("./modelQ")
sys.path.append("./dynamicsQ")
'''
sys.path.append("./sample_data")
from dataloader import *
train_data = np.load('/content/sample_data/train_dataset.npz')
train_inputs = train_data["input"]
train_labels = train_data["labels"]
X_inputs = train_inputs[:100]#50000]
y_labels = train_labels[:100]#50000]
#train_dataset = DynamicsDataset(train_inputs, train_labels)
train_dataset = DynamicsDataset(X_inputs, y_labels)

train_dataloader = torch.utils.data.DataLoader(train_dataset,
                                                 batch_size=1,
                                                 shuffle=True,
                                                 collate_fn=DynamicsDataset.collate_fn,
                                                 pin_memory=True,
                                                 num_workers=1)
print(len(X_inputs))
print(len(y_labels))

```

    100
    100

```python
from qiskit import transpile, assemble
class QuanvCircuit:
    """
    This class defines filter circuit of Quanvolution layer
    """
    def __init__(self, kernel_size, backend, shots, threshold):
        # --- Circuit definition start ---
        self.n_qubits = kernel_size ** 2
        self._circuit = qiskit.QuantumCircuit(self.n_qubits)
        self.theta = [qiskit.circuit.Parameter('theta{}'.format(i)) for i in range(self.n_qubits)]

        for i in range(self.n_qubits):
            self._circuit.rx(self.theta[i], i)

        #self._circuit.barrier()
        self._circuit.measure_all()
        # ---- Circuit definition end ----
        self.backend   = backend
        self.shots     = shots
        self.threshold = threshold

    def run(self, data):
        # encoding data to parameters
        thetas = [data]

        param_dict = dict()
        for theta in thetas:
            for i in range(self.n_qubits):
                param_dict[self.theta[i]] = theta[i]
        param_binds = [param_dict]

        List = [thetas]
        t_qc = transpile(self._circuit,
                         self.backend)
        qobj = assemble(t_qc,
                        shots=self.shots,
                        parameter_binds = param_binds)
        job = self.backend.run(qobj)
        result = job.result().get_counts()

        # decoding the result
        counts = 0
        for key, val in result.items():
            cnt = sum([int(char) for char in key])
            counts += cnt * val

        probabilities = counts / (self.shots * self.n_qubits)

        return probabilities
```

```python
backend = qiskit.Aer.get_backend('qasm_simulator')
filter_size = 2
circ = QuanvCircuit(filter_size, backend, 100, 127)
#data = torch.tensor([[0, 200], [100, 255]])
#data = torch.tensor([-2.8351264, 1.3314428,  -0.18492305,  0.00792312]).numpy()#1,2,3,4]).numpy()
data = [-2.8351264,  1.3314428,  -0.18492305,  0.00792312]

#
print(data)
#print(data.size())
print(circ.run(data))

circ._circuit.draw()#output='mpl')
```

    [-2.8351264, 1.3314428, -0.18492305, 0.00792312]
    0.3325

<pre style="word-wrap: normal;white-space: pre;background: #fff0;line-height: 1.1;font-family: "Courier New",Courier,monospace">        ┌────────────┐ ░ ┌─┐
   q_0: ┤ Rx(theta0) ├─░─┤M├─────────
        ├────────────┤ ░ └╥┘┌─┐
   q_1: ┤ Rx(theta1) ├─░──╫─┤M├──────
        ├────────────┤ ░  ║ └╥┘┌─┐
   q_2: ┤ Rx(theta2) ├─░──╫──╫─┤M├───
        ├────────────┤ ░  ║  ║ └╥┘┌─┐
   q_3: ┤ Rx(theta3) ├─░──╫──╫──╫─┤M├
        └────────────┘ ░  ║  ║  ║ └╥┘
meas: 4/══════════════════╩══╩══╩══╩═
                          0  1  2  3 </pre>

```python
import qiskit
backend = qiskit.Aer.get_backend('qasm_simulator')
filter_size = 2
circ = QuanvCircuit(filter_size, backend, 100, 127)
```

```python
#for (X_train, y_train) in train_dataloader:
print('Loading Training data')
xVal = []
yVal = []
#count = 0
for (X_train, y_train) in train_dataloader:
    #print('Training Starts')
    #print(X_train.detach().numpy()[0])
    #print(y_train.detach().numpy()[0])
    x = circ.run(X_train.detach().numpy()[0])
    y = circ.run(y_train.detach().numpy()[0])
    #regressor.fit(x, y)

    xVal.append([x])
    yVal.append(y)
    '''
    count += 1

    if (count == 50000):
        print('Loaded data :',count)
        break
    '''
print('x data',len(xVal))
print('y data',len(yVal))

```

    Loading Training data

    ./sample_data/dataloader.py:35: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      batch_x = torch.as_tensor(batch_x)

    x data 100
    y data 100

```python
#print(torch.tensor(xVal).detach().numpy())
#print(torch.tensor(yVal).detach().numpy())
import time
print('Training Started')
startTime = time.time()
regressor.fit(torch.tensor(xVal).detach().numpy(), torch.tensor(yVal).detach().numpy())
print('Total Training Time :', (time.time()-startTime))
```

```python
#print(torch.tensor(xVal).detach().numpy())
#print(torch.tensor(yVal).detach().numpy())
import time
print('Training Started')
startTime = time.time()
regressor_noise.fit(torch.tensor(xVal).detach().numpy(), torch.tensor(yVal).detach().numpy())
print('Total Training Time :', (time.time()-startTime))
```

```python
# score the result
regressor.score(torch.tensor(xVal).detach().numpy(), torch.tensor(yVal).detach().numpy())
```

    -0.8868040210846102

```python
# score the result
regressor_noise.score(torch.tensor(xVal).detach().numpy(), torch.tensor(yVal).detach().numpy())
```

    -0.8087757357649599

```python
import matplotlib.pyplot as plt

plt.style.use("seaborn")
fig, (ax1) = plt.subplots(figsize=(6,6))

ax1.plot(objective_func_vals, "-ob", label="Response without noise")
ax1.plot(objective_func_vals_noise, "-or", label="Response with noise")

ax1.set_ylabel("Objective Function Value")
ax1.set_ylim([2, 45])
ax1.set_xlabel("Iteration")
ax1.legend()#loc=2, prop = {'size': 10})
```


# **Save this file as studentid1_studentid2_lab#.ipynb**
(Your student-id is the number shown on your student card.)

E.g. if you work with 3 people, the notebook should be named:
12301230_3434343_1238938934_lab1.ipynb.

**This will be parsed by a regexp, so please double check your filename.**

Before you turn this problem in, please make sure everything runs correctly. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All). Note, that **you are not allowed to use Google Colab**.

**Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your names and email adresses below.**

```python
NAME = "Radu Sibechi"
NAME2 = "Nicu-Traian Vidrascu"
EMAIL = "ntvidrascu@gmail.com"
EMAIL2 = "radu.sibechi@yahoo.com"

```

# Lab 3: Gaussian Processes and Support Vector Machines

### Machine Learning 1, November 2018

Notes on implementation:

* You should write your code and answers in this IPython Notebook: http://ipython.org/notebook.html. If you have problems, please contact your teaching assistant.
* Please write your answers right below the questions.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Refer to last week's lab notes, i.e. http://docs.scipy.org/doc/, if you are unsure about what function to use. There are different correct ways to implement each problem!
* use the provided test boxes to check if your answers are correct

```python
%pylab inline
plt.rcParams["figure.figsize"] = [20,10]
```

    Populating the interactive namespace from numpy and matplotlib

```python
# This cell makes sure that you have all the necessary libraries installed

import sys
import platform
from importlib.util import find_spec, module_from_spec

def check_newer_version(version_inst, version_nec):
    version_inst_split = version_inst.split('.')
    version_nec_split = version_nec.split('.')
    for i in range(min(len(version_inst_split), len(version_nec_split))):
        if int(version_nec_split[i]) > int(version_inst_split[i]):
            return False
        elif int(version_nec_split[i]) < int(version_inst_split[i]):
            return True
    return True

module_list = [('jupyter', '1.0.0'),
               ('matplotlib', '2.0.2'),
               ('numpy', '1.13.1'),
               ('python', '3.6.2'),
               ('sklearn', '0.19.0'),
               ('scipy', '0.19.1'),
               ('nb_conda', '2.2.1')]

packages_correct = True
packages_errors = []

for module_name, version in module_list:
    if module_name == 'scikit-learn':
        module_name = 'sklearn'
    if module_name == 'pyyaml':
        module_name = 'yaml'
    if 'python' in module_name:
        python_version = platform.python_version()
        if not check_newer_version(python_version, version):
            packages_correct = False
            error = f'Update {module_name} to version {version}. Current version is {python_version}.'
            packages_errors.append(error)
            print(error)
    else:
        spec = find_spec(module_name)
        if spec is None:
            packages_correct = False
            error = f'Install {module_name} with version {version} or newer, it is required for this assignment!'
            packages_errors.append(error)
            print(error)
        else:
            x = __import__(module_name)
            if hasattr(x, '__version__') and not check_newer_version(x.__version__, version):
                packages_correct = False
                error = f'Update {module_name} to version {version}. Current version is {x.__version__}.'
                packages_errors.append(error)
                print(error)

try:
    from google.colab import drive
    packages_correct = False
    error = """Please, don't use google colab!
It will make it much more complicated for us to check your homework as it merges all the cells into one."""
    packages_errors.append(error)
    print(error)
except:
    pass

packages_errors = '\n'.join(packages_errors)
```

    Install nb_conda with version 2.2.1 or newer, it is required for this assignment!

## Part 1: Gaussian Processes

For part 1 we will be refer to Bishop sections 6.4.2 and 6.4.3. You may also want to refer to Rasmussen's Gaussian Process text which is available online at http://www.gaussianprocess.org/gpml/chapters/ and especially to the project found at https://www.automaticstatistician.com/index/ by Ghahramani for some intuition in GP.  To understand Gaussian processes, it is highly recommended understand how marginal, partitioned Gaussian distributions can be converted into conditional Gaussian distributions.  This is covered in Bishop 2.3 and summarized in Eqns 2.94-2.98.
$\newcommand{\bt}{\mathbf{t}}$
$\newcommand{\bx}{\mathbf{x}}$
$\newcommand{\by}{\mathbf{y}}$
$\newcommand{\bw}{\mathbf{w}}$
$\newcommand{\ba}{\mathbf{a}}$

### Periodic Data

We will use the same data generating function that we used previously for regression.

```python
def true_mean_function(x):
    return np.sin(2*pi*(x+1))

def add_noise(y, sigma):
    return y + sigma*np.random.randn(len(y))

def generate_t(x, sigma):
    return add_noise(true_mean_function(x), sigma)

```

```python
sigma = 0.2
beta  = 1.0 / pow(sigma, 2)
N_test = 100

x_test = np.linspace(-1, 1, N_test)
mu_test = np.zeros(N_test)
y_test = true_mean_function(x_test)
t_test = add_noise(y_test, sigma)

plt.plot( x_test, y_test, 'b-', lw=2)
plt.plot( x_test, t_test, 'go')
plt.show()
```

### 1. Sampling from the Gaussian process prior (30 points)
We will implement Gaussian process regression using the kernel function in Bishop Eqn. 6.63.

#### 1.1 Why kernel function? (5 points)

Before implementing kernel function, it would be useful to understand why kernels are crucial for high dimensional data. Why are kernels useful when you are dealing with high dimensional data?

YOUR ANSWER HERE
Kernels are usefull for making calculations in a lower dimension than the original one, thus they decrease complexity.

#### 1.2 k_n_m( xn, xm, thetas ) (5 points)
To start, implement function `k_n_m(xn, xm, thetas)` that takes scalars $x_n$ and $x_m$, and a vector of $4$ thetas, and computes the kernel function Bishop Eqn. 6.63 (10 points).  NB: usually the kernel function will take $D$ by $1$ vectors, but since we are using a univariate problem, this makes things easier.

```python
def k_n_m(xn, xm, thetas):
    k = thetas[0]*np.exp(-thetas[1]/2*np.dot((xn-xm).T,(xn-xm)))+thetas[2]+thetas[3]*xn.dot(xm)
    return k
```

```python
# DO NOT REMOVE THIS CELL!
# It contains hidden tests

```

#### 1.3 computeK( X1, X2, thetas ) (10 points)
Eqn 6.60 is the marginal distribution of mean output of $N$ data vectors: $p(\mathbf{y}) = \mathcal{N}(0, \mathbf{K})$.  Notice that the expected mean function is $0$ at all locations, and that the covariance is a $N$ by $N$ kernel matrix $\mathbf{K}$.  Write a function `computeK(x1, x2, thetas)`
that computes the kernel matrix. Use k_n_m as part of an inner loop (of course, there are more efficient ways of computing the kernel function making better use of vectorization, but that is not necessary) (5 points).

```python
def computeK(x1, x2, thetas):
    if not type(x1) is np.ndarray or len(x1.shape)==1:
        #the case when X has shape is Nx1
        x1 = np.array(x1)
        x2 = np.array(x2)
        diff = x1[:,np.newaxis] - x2
        x1 = np.array([x1]).T
        x2 = np.array([x2]).T
        diff_norm = diff*diff
    else:
        #the case when X has shape NxD
        diff = x1[:,np.newaxis] - x2
        diff_norm = sum(diff*diff,axis=-1)

    K = thetas[0]*np.exp(-thetas[1]/2*diff_norm)+thetas[2]+thetas[3]*np.dot(x1,x2.T)
    return K
```

```python
### Test your function
x1 = [0, 1, 2]
x2 = [1, 2, 3, 4]
thetas = [1, 2, 3, 4]
K = computeK(x1, x2, thetas)

assert K.shape == (len(x1), len(x2)), "the shape of K is incorrect"

```

#### 1.4 Plot function samples (15 points)
Now sample mean functions at the x_test locations for the theta values in Bishop Figure 6.5, make a figure with a 2 by 3 subplot and make sure the title reflects the theta values (make sure everything is legible).  In other words, sample $\by_i \sim \mathcal{N}(0, \mathbf{K}_{\theta})$.  Make use of numpy.random.multivariate_normal().  On your plots include the expected value of $\by$ with a dashed line and fill_between 2 standard deviations of the uncertainty due to $\mathbf{K}$ (the diagonal of $\mathbf{K}$ is the variance of the model uncertainty) (15 points).

```python
# YOUR CODE HERE
def plot_function_samples():
    thetas = [[1,4,0,0],[9,4,0,0],[1,64,0,0],[1,25,0,0],[1,4,10,0],[1,4,0,5]]
    num_cols = 3
    num_rows = 2
    for i,theta in enumerate(thetas):
        ax = plt.subplot(num_rows, num_cols, i+1)
        for _ in range(1):
            K = computeK(x_test,x_test,theta)
            plt.plot(x_test,np.random.multivariate_normal([0]*100,K))

            plt.errorbar(x_test,y_test,2*np.sqrt(K[range(len(x_test)),range(len(x_test))]))
        plt.title(theta)
    plt.tight_layout()
    plt.show()

plot_function_samples()

```

### 2. Predictive distribution (35 points)
So far we have sampled mean functions from the prior.  We can draw actual data $\bt$ two ways.  The first way is generatively, by first sampling $\by | \mathbf{K}$, then sampling $\bt | \by, \beta$ (Eqns 6.60 followed by 6.59).  The second way is to integrate over $\by$ (the mean draw) and directly sample $\bt | \mathbf{K}, \beta$ using Eqn 6.61.    This is the generative process for $\bt$.  Note that we have not specified a distribution over inputs $\bx$;  this is because Gaussian processes are conditional models.  Because of this we are free to generate locations $\bx$ when playing around with the GP; obviously a dataset will give us input-output pairs.

Once we have data, we are interested in the predictive distribution (note: the prior is the predictive distribution when there is no data).  Consider the joint distribution for $N+1$ targets, given by Eqn 6.64.  Its covariance matrix is composed of block components $C_N$, $\mathbf{k}$, and $c$.  The covariance matrix $C_N$ for $\bt_N$ is $C_N = \mathbf{K}_N + \beta^{-1}\mathbf{I}_N$.  We have just made explicit the size $N$ of the matrix; $N$ is the number of training points.  The kernel vector $\mathbf{k}$ is a $N$ by $1$ vector of kernel function evaluations between the training input data and the test input vector.  The scalar $c$ is a kernel evaluation at the test input.

#### 2.1 gp_predictive_distribution(...) (10 points)
Write a function `gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C=None)` that computes  Eqns 6.66 and 6.67, except allow for an arbitrary number of test points (not just one) and now the kernel matrix is for training data.  By having C as an optional parameter, we can avoid computing it more than once (for this problem it is unimportant, but for real problems this is an issue).  The function should compute $\mathbf{C}$, $\mathbf{k}$, and return the mean, variance and $\mathbf{C}$.  Do not forget: the computeK function computes $\mathbf{K}$, not $\mathbf{C}$.(10 points)

```python
def gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C=None):
    K = computeK(x_train,x_train,theta)
    if C is None:
        C = K+1/beta*np.eye(K.shape[0])
    c = computeK(x_test,x_test,theta) + 1/beta
    k = computeK(x_train,x_test,theta)
    kTC1 = np.dot(k.T,np.linalg.inv(C))
    mean_test = kTC1.dot(t_train)
    covar_test = c - kTC1.dot(k)

    return mean_test, covar_test, C
```

```python
### Test your function
N = 2
train_x = np.linspace(-1, 1, N)
train_t = 2*train_x
test_N = 3
test_x = np.linspace(-1, 1, test_N)
theta = [1, 2, 3, 4]
beta = 25
test_mean, test_covar, C = gp_predictive_distribution(train_x, train_t, test_x, theta, beta, C=None)

assert test_mean.shape == (test_N,), "the shape of mean is incorrect"
assert test_covar.shape == (test_N, test_N), "the shape of var is incorrect"
assert C.shape == (N, N), "the shape of C is incorrect"

C_in = np.array([[0.804, -0.098168436], [-0.098168436, 0.804]])
_, _, C_out = gp_predictive_distribution(train_x, train_t, test_x, theta, beta, C=C_in)

assert np.allclose(C_in, C_out), "C is not reused!"

```

#### 2.2 gp_log_likelihood(...) (10 points)
To learn the hyperparameters, we would need to compute the log-likelihood of the of the training data.  Implicitly, this is conditioned on the value setting for $\mathbf{\theta}$.  Write a function `gp_log_likelihood(x_train, t_train, theta, C=None, invC=None, beta=None)`, where C and invC can be stored and reused. It should return the log-likelihood, `C` and `invC`  (10 points)

```python
def gp_log_likelihood(x_train, t_train, theta, beta, C=None, invC=None):
    N = t_train.shape[0]
    if C is None or invC is None:
        K = computeK(x_train,x_train,theta)

        if C is None:
            C = K+1/beta*np.eye(K.shape[0])

        if invC is None:
            invC = np.linalg.inv(C)

    lp = -1/2*np.log(np.abs(np.sum((C))))- 1/2*t_train.T.dot(invC).dot(t_train) - N/2*np.log(2*np.pi)

    return lp, C, invC
```

```python
### Test your function
N = 2
train_x = np.linspace(-1, 1, N)
train_t = 2 * train_x
theta = [1, 2, 3, 4]
beta = 25
lp, C, invC = gp_log_likelihood(train_x, train_t, theta, beta, C=None, invC=None)

assert lp < 0, "the log-likelihood should be smaller than 0"
assert C.shape == (N, N), "the shape of var is incorrect"
assert invC.shape == (N, N), "the shape of C is incorrect"

C_in = np.array([[0.804, -0.098168436], [-0.098168436, 0.804]])
_, C_out, _ = gp_log_likelihood(train_x, train_t, theta, beta, C=C_in, invC=None)

assert np.allclose(C_in, C_out), "C is not reused!"

invC_in = np.array([[1.26260453, 0.15416407], [0.15416407, 1.26260453]])
_, _, invC_out = gp_log_likelihood(train_x, train_t, theta, beta, C=None, invC=invC_in)

assert np.allclose(invC_in, invC_out), "invC is not reused!"

```

#### 2.3 Plotting (10 points)
Repeat the 6 plots above, but this time conditioned on the training points.  Use the periodic data generator to create 2 training points where x is sampled uniformly between $-1$ and $1$.  For these plots, feel free to use the provided function "gp_plot".  Make sure you put the parameters in the title and this time also the log-likelihood. Try to understand the two types of uncertainty!  If you do not use `gp_plot(...)`, please add a fill between for the model and target noise. (10 points)

```python
def gp_plot( x_test, y_test, mean_test, covar_test, x_train, t_train, theta, beta ):
    # x_test:
    # y_test:    the true function at x_test
    # mean_test: predictive mean at x_test
    # var_test:  predictive covariance at x_test
    # t_train:   the training values
    # theta:     the kernel parameters
    # beta:      the precision (known)

    # the reason for the manipulation is to allow plots separating model and data stddevs.
    std_total = np.sqrt(np.diag(covar_test))       # includes all uncertainty, model and target noise
    std_model = np.sqrt(std_total**2 - 1.0/beta) # remove data noise to get model uncertainty in stddev
    std_combo = std_model + np.sqrt(1.0/beta)    # add stddev (note: not the same as full)

    plt.plot(x_test, y_test, 'b', lw=3)
    plt.plot(x_test, mean_test, 'k--', lw=2)
    plt.fill_between(x_test, mean_test+2*std_combo,mean_test-2*std_combo, color='k', alpha=0.25)
    plt.fill_between(x_test, mean_test+2*std_model,mean_test-2*std_model, color='r', alpha=0.25)
    plt.plot(x_train, t_train, 'ro', ms=10)

```

```python
def plot_function_samples2(thetas = None, sigma = None, beta = None, N_train = None, N_test = None):
    if thetas is None:
        thetas = [[1,4,0,0],[9,4,0,0],[1,64,0,0],[1,25,0,0],[1,4,10,0],[1,4,0,5]]
    if sigma is None:
        sigma = 0.2
    if beta is None:
        beta = 1.0 / pow(sigma, 2)
    if N_train is None:
        N_train = 2
    if N_test is None:
        N_test = 100

    x = np.random.uniform(-1,1,N_train)
    t_train = generate_t(x,sigma)

    x_test = np.linspace(-1, 1, N_test)
    mu_test = np.zeros(N_test)
    y_test = true_mean_function(x_test)
    t_test = add_noise(y_test, sigma)

    num_cols = 3
    num_rows = 2
    for i,theta in enumerate(thetas):
        ax = plt.subplot(num_rows, num_cols, i+1)
        mean, var, C = gp_predictive_distribution(x,t_train,x_test,theta,beta)
        gp_plot(x_test,y_test,mean,var,x,t_train,theta,beta)
        lp, _, _ = gp_log_likelihood(x,t_train,theta,beta)
        plt.title(str(theta) + ' ' + str(lp))
    plt.tight_layout()
    plt.show()
plot_function_samples2(N_train = 100)
```

#### 2.4 More plotting (5 points)
Repeat the 6 plots above, but this time conditioned a new set of 10 training points. (5 points)

```python
plot_function_samples2(N_train = 10)
```

## Part 2: Support Vector Machines (45 points)
As seen in Part 1: Gaussian Processes, one of the significant limitations of many such algorithms is that the kernel function $k(\bx_n , \bx_m)$ must be evaluated for all possible pairs $\bx_n$ and $\bx_m$ of training points, which can be computationally infeasible during training and can lead to excessive computation times when making predictions for new data points.
In Part 2: Support Vector Machines, we shall look at kernel-based algorithms that have sparse solutions, so that predictions for new inputs depend only on the kernel function evaluated at a subset of the training data points. We are using the same notation as in Bishop chapter 7.

### 2.1 Generating a linearly separable dataset (15 points)
a) (5 points) First of all, we are going to create our own 2D toy dataset $X$. The dataset will consists of two i.i.d. subsets $X_1$ and $X_2$, each of the subsets will be sampled from a multivariate Gaussian distribution,

\begin{align}
X_1 \sim &\mathcal{N}(\mu_1, \Sigma_1)\\
&\text{ and }\\
X_2 \sim &\mathcal{N}(\mu_2, \Sigma_2).
\end{align}

In the following, $X_1$ will have $N_1=20$ samples and a mean $\mu_1=(1,1)$. $X_2$ will have $N_2=30$ samples and a mean $\mu_2=(3,3)$.

Plot the two subsets in one figure, choose two colors to indicate which sample belongs to which subset. In addition you should choose, $\Sigma_1$ and $\Sigma_2$ in a way that the two subsets become linearly separable. (Hint: Which form has the covariance matrix for a i.i.d. dataset?)

```python
def generate_XX(N1 = None, N2 = None):
    if N1 is None:
        N1 = 20
        N2 = 30
    X1 = np.random.multivariate_normal([1,1],[[0.1, 0],[0, 0.1]], N1)
    X2 = np.random.multivariate_normal([3,3],[[0.2, 0],[0, 0.2]], N2)
    return X1, X2

def plot_XX(X1,X2):
    plt.plot(X1[:,0],X1[:,1],'go')
    plt.plot(X2[:,0],X2[:,1],'bo')

X1, X2 = generate_XX()
plot_XX(X1, X2)
N1 = X1.shape[0]
N2 = X2.shape[0]
```

b) (10 points) In the next step we will combine the two datasets X_1, X_2 and generate a vector `t` containing the labels. Write a function `create_X_and_t(X1, X2)` it should return the combined data set X and the corresponding target vector t.

```python
def create_X_and_t(X1, X2):
    t1 = np.ones((X1.shape[0],))
    t2 = -np.ones((X2.shape[0]),)
    t = np.concatenate((t1,t2))
    X = np.vstack((X1,X2))
    return X, t
```

```python
### Test your function
dim = 2
N1_test = 3
N2_test = 4
X1_test = np.arange(6).reshape((N1_test, dim))
X2_test = np.arange(8).reshape((N2_test, dim))
X_test, t_test = create_X_and_t(X1_test, X2_test)

assert X_test.shape == (N1_test + N2_test, dim), "the shape of X is incorrect"
assert t_test.shape == (N1_test + N2_test,), "the shape of t is incorrect"

```

### 2.2 Finding the support vectors (15 points)
Finally we going to use a SVM to obtain the decision boundary for which the margin is maximized. We have to solve the optimization problem

\begin{align}
\arg \min_{\bw, b} \frac{1}{2} \lVert \bw \rVert^2,
\end{align}

subject to the constraints

\begin{align}
t_n(\bw^T \phi(\bx_n) + b) \geq 1, n = 1,...,N.
\end{align}

In order to solve this constrained optimization problem, we introduce Lagrange multipliers $a_n \geq 0$. We obtain the dual
representation of the maximum margin problem in which we maximize

\begin{align}
\sum_{n=1}^N a_n - \frac{1}{2}\sum_{n=1}^N\sum_{m=1}^N a_n a_m t_n t_m k(\bx_n, \bx_m),
\end{align}

with respect to a subject to the constraints

\begin{align}
a_n &\geq 0, n=1,...,N,\\
\sum_{n=1}^N a_n t_n &= 0.
\end{align}

This takes the form of a quadratic programming problem in which we optimize a quadratic function of $\mathbf{a}$, subject to a set of inequality constraints.

a) (5 points) In this example we will use a linear kernel $k(\bx, \bx') = \bx^T\bx'$. Write a function `computeK(X)` that computes the kernel matrix $K$ for the 2D dataset $X$.

```python
def computeK(X):
    K = np.dot(X,X.T)

    return K
```

```python
dim = 2
N_test = 3
X_test = np.arange(2, 8).reshape((N_test, dim))
K_test = computeK(X_test)

assert K_test.shape == (N_test, N_test)

```

Next, we will rewrite the dual representation so that we can make use of computationally efficient vector-matrix multiplication. The objective becomes

\begin{align}
\min_{\ba} \frac{1}{2} \ba^T K' \ba - 1^T\ba,
\end{align}

subject to the constraints

\begin{align}
a_n &\geq 0, n=1,...,N,\\
\bt^T\ba &= 0.
\end{align}

Where
\begin{align}
K'_{nm} = t_n t_m k(\bx_n, \bx_m),
\end{align}
and in the special case of a linear kernel function,
\begin{align}
K'_{nm} = t_n t_m k(\bx_n, \bx_m) = k(t_n \bx_n, t_m \bx_m).
\end{align}

To solve the quadratic programming problem we will use a python module called cvxopt. You first have to install the module in your virtual environment (you have to activate it first), using the following command:

`conda install -c anaconda cvxopt`

The quadratic programming solver can be called as

`cvxopt.solvers.qp(P, q[, G, h[, A, b[, solver[, initvals]]]])`

This solves the following problem,

\begin{align}
\min_{\bx} \frac{1}{2} \bx^T P \bx + q^T\bx,
\end{align}

subject to the constraints,

\begin{align}
G\bx &\leq h,\\
A\bx &= b.
\end{align}

All we need to do is to map our formulation to the cvxopt interface.

b) (10 points) Write a function `compute_multipliers(X, t)` that solves the quadratic programming problem using the cvxopt module and returns the lagrangian multiplier for every sample in the dataset.

```python
import cvxopt

def compute_multipliers(X, t):
    K = computeK(X*t[:,np.newaxis])
    q = cvxopt.matrix(-1*np.ones((X.shape[0]),dtype=np.double))
    A = cvxopt.matrix(t[:,np.newaxis].T.astype(np.double))
    b = cvxopt.matrix(np.array([0]).astype(np.double))
    G = cvxopt.matrix(-1*np.eye(X.shape[0],X.shape[0],dtype=np.double))
    h = cvxopt.matrix(np.zeros_like(t.astype(np.double)))
    P = cvxopt.matrix(K.astype(np.double))

    sol = cvxopt.solvers.qp(P, q, G, h, A, b)
    a = np.array(sol['x'])
    return a
```

```python
### Test your function
dim = 2
N_test = 3
X_test = np.arange(2, 8).reshape((N_test, dim))
t_test = np.array([-1., 1., 1.])
a_test = compute_multipliers(X_test, t_test)

assert a_test.shape == (N_test, 1)

```

### 2.3 Plot support vectors (5 points)
Now that we have obtained the lagrangian multipliers $\ba$, we use them to find our support vectors. Repeat the plot from 2.1, this time use a third color to indicate which samples are the support vectors.

```python
def plot_X_SV(X, N1, N2, a):
    G = -1*np.eye(X.shape[0],X.shape[0],dtype=np.double)
    h = np.zeros_like(X.shape[0]).astype(np.double)
    plt.plot(X[0:N1,0],X[0:N1,1],'co')
    plt.plot(X[N1:N2,0],X[N1:N2,1],'yo')
    srt = argsort(a[:,0])

    for arg in srt:
        if a[arg]>1e-5:
            #plt.plot(,facecolors='none', edgecolors='r')
            plt.scatter(X[arg,0],X[arg,1], s=240, facecolors='none', edgecolors='purple')

X, t = create_X_and_t(X1, X2)
a = compute_multipliers(X, t)
srt = argsort(a[:,0])
supports = []
for i, val in enumerate(a):
    if val>1e-5:
        supports.append(i)

plot_X_SV(X, N1, N2, a)
```

### 2.4 Plot the decision boundary (10 Points)
The decision boundary is fully specified by a (usually very small) subset of training samples, the support vectors. Make use of

\begin{align}
\bw &= \sum_{n=1}^N a_n t_n \mathbf{\phi}(\bx_n)\\
b &= \frac{1}{N_S}\sum_{n \in S} (t_n - \sum_{m \in S} a_m t_m k(\bx_n, \bx_m)),
\end{align}

where $S$ denotes the set of indices of the support vectors, to calculate the slope and intercept of the decision boundary. Generate a last plot that contains the two subsets, support vectors and decision boundary.

```python
def plot_X_SV_DB(X, a):
    G = -1*np.eye(X.shape[0],X.shape[0],dtype=np.double)
    h = np.zeros_like(X.shape[0]).astype(np.double)
    plt.plot(X[0:N1,0],X[0:N1,1],'co')
    plt.plot(X[N1:N2,0],X[N1:N2,1],'yo')
    srt = argsort(a[:,0])

    for arg in srt:
        if a[arg]>1e-5:
            #plt.plot(,facecolors='none', edgecolors='r')
            plt.scatter(X[arg,0],X[arg,1], s=240, facecolors='none', edgecolors='purple')

    w = np.sum ( a * t[:, np.newaxis] * X, axis = 0 )

    b = t[supports] - np.dot(X[supports], w)
    bias = b[0]
    slope = -w[0] / w[1]
    intercept = -bias / w[1]
    x = np.arange(0, 6)

    margin = X[supports][1][0] * slope + intercept - X[supports][1][1]
    plt.plot(x, x * slope + intercept, 'r-')
    plt.plot(x, x * slope + intercept + margin, 'b-')
    plt.plot(x, x * slope + intercept - margin, 'b-')

plot_X_SV_DB(X,a)
```


```python
from IPython.core.display import HTML
def css_styling():
    styles = open("./styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: #888;
    color: #f6f6f6;
}

div #notebook { /* centre the content */
    background: #fff;
    color: #333;
    width: 1200px;
    margin: auto;
    padding-left: 1em;
    padding-right: 1em;
    padding-top: 2ex;
    overflow-x: auto;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

div.cell { /* set cell width to about 80 chars */
    width: 900px;
}

div.cell.border-box-sizing.code_cell.running {
    /* draw border around running cells */
    border: 3px solid #111;
}

h1 {
    font-family: 'Philosopher', sans-serif;
}
h2 {
    font-family: 'Philosopher', serif;
}
h3{
    font-family: 'Philosopher', serif;
    font-style: 'italic';
    margin-top:12px;
    margin-bottom: 3px;
}
h4{
    font-family: 'Philosopher', serif;
}
h5 {
    font-family: 'Alegreya Sans', sans-serif;
}
h6 {
    font-family: 'PT Mono', sans-serif;
}

div.text_cell_render{
    font-family: 'Arvo' sans-serif;
    line-height: 130%;
    font-size: 115%;
    width:800px;
    margin-left:auto;
    margin-right:auto;
}

.CodeMirror{
        font-family: "PT Mono";
                    font-size: 100%;
            }

.text_cell_render h1 {
    font-weight: 400;
    font-size: 64pt;
    line-height: 100%;
    color: rgb(12,85,97);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}
.text_cell_render h2 {
    font-weight: 700;
    font-size: 24pt;
    line-height: 100%;
    color: rgb(171,165,131);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h3 {
    font-style: italic;
    color: rgb(95,92,72);
}

.text_cell_render h5 {
    font-weight: 300;
    font-size: 16pt;
    color: grey;
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h6 {
    font-weight: 300;
    font-size: 10pt;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}
div.cell.code_cell {  /* Group area containing both code and output */
background-color: rgba(171,165,131,0.3);
border-radius: 10px; /* rounded borders */
padding: 1em;
}

</style>

# Approximate solutions to the Riemann Problem

## Solutions in practice

Solutions to the Riemann problem are mainly used in two contexts:

1. As reference solutions against which a numerical method is benchmarked, or
2. As part of a numerical method, such as a high resolution shock capturing method, where the flux between two numerical cells is required.

In the first case, accuracy is paramount and the complete solution (all wave speeds, and all intermediate states) is required. In the second case only one thing is required: the flux ${\bf f}^*$ between the cells, which is the flux on the characteristic line $\xi = x / t = 0$.

In this second case, the numerical method will have to repeatedly solve the Riemann problem. In a general problem, the solution may be needed tens of times *per cell, per timestep*, leading to millions (or more!) solutions in a simulation. The speed of the solution is then extremely important, and approximate solutions are often used.

## Roe-type solutions

The most obvious simplification is to reduce the nonlinear problem

\begin{equation}
  \partial_t {\bf q} + \partial_x {\bf f}({\bf q}) = {\bf 0}
\end{equation}

to the *linear* problem

\begin{equation}
  \partial_t {\bf q} + A \partial_x {\bf q} = {\bf 0},
\end{equation}

where $A$ is a *constant* matrix that approximates the Jacobian $\partial {\bf f} / \partial {\bf q}$. We can then solve the linear problem exactly (e.g. by diagonalising the matrix and solving the resulting uncoupled advection equations), to find

\begin{align}
  {\bf q}(x, t) & = {\bf q}_l + \sum_{p: \lambda^{(p)} < \tfrac{x}{t}} \left\{ {\bf l}^{(p)} \cdot \left( {\bf q}_r - {\bf q}_l \right) \right\} {\bf r}^{(p)}, \\
   & = {\bf q}_r - \sum_{p: \lambda^{(p)} > \tfrac{x}{t}} \left\{ {\bf l}^{(p)} \cdot \left( {\bf q}_r - {\bf q}_l \right) \right\} {\bf r}^{(p)}, \\
   & = \frac{1}{2} \left( {\bf q}_l + {\bf q}_r \right) + \sum_{p: \lambda^{(p)} < \tfrac{x}{t}} \left\{ {\bf l}^{(p)} \cdot \left( {\bf q}_r - {\bf q}_l \right) \right\} {\bf r}^{(p)} - \sum_{p: \lambda^{(p)} > \tfrac{x}{t}} \left\{ {\bf l}^{(p)} \cdot \left( {\bf q}_r - {\bf q}_l \right) \right\} {\bf r}^{(p)}.
\end{align}

where $\lambda^{(p)}, {\bf r}^{(p)},$ and ${\bf l}^{(p)}$ are the eigenvalues and the (right and left respectively) eigenvectors of $A$, ordered such that $\lambda^{(1)} \le \dots \le \lambda^{(N)}$ as usual. All three solutions are equivalent; the last is typically used.

Given this complete solution, it is easily evaluated along $x = 0$, and the flux calculated from the result.

An even greater shortcut can be found by noting that we are approximating ${\bf f} = A {\bf q}$. Therefore the standard form is to write

\begin{equation}
  {\bf f}^* = \frac{1}{2} \left( {\bf f}_l + {\bf f}_r \right) + \sum_{p} \left| \lambda^{(p)} \right| \left\{ {\bf l}^{(p)} \cdot \left( {\bf q}_r - {\bf q}_l \right) \right\} {\bf r}^{(p)},
\end{equation}

where now we are summing over all eigenvalues and eigenvectors. It should be noted that ${\bf f}^* \ne {\bf f}({\bf q}^*)$ in general, as the calculation of ${\bf f}^*$ relied on an approximation to the flux.

In order to complete this specification of the solver, we only need to say how $A$ is defined. Roe gave the suggestion that

\begin{equation}
  A = A({\bf q}_{\textrm{Roe}}) = \left. \frac{\partial {\bf f}}{\partial {\bf q}} \right|_{{\bf q}_{\textrm{Roe}}},
\end{equation}

where the *Roe average* ${\bf q}_{\textrm{Roe}}$ satisfies

1. $A({\bf q}_{\textrm{Roe}}) \left( {\bf q}_r - {\bf q}_l \right) = {\bf f}_r - {\bf f}_l$,
2. $A({\bf q}_{\textrm{Roe}})$ is diagonalizable with real eigenvalues, and
3. $A({\bf q}_{\textrm{Roe}}) \to \partial {\bf f} / \partial {\bf q}$ smoothly as ${\bf q}_{\textrm{Roe}} \to {\bf q}$.

It is *possible* to construct the Roe average for many systems (such as the Euler equations, and the relativistic Euler equations). However, a simple arithmetic average is often nearly as good - in the sense that the algorithm will fail only slightly more often than the algorithm with the full Roe average!

The problem with Roe type solvers is that it approximates all waves as discontinuities. This leads to inaccuracies near rarefactions, and these can be catastrophically bad when the rarefaction fan crosses $\xi = 0$ (a *sonic rarefaction*). It is possible to detect when these problems will occur (e.g. by looking at when $\lambda^{(p)}$ changes sign between the left and right states) and change the approximation at this point, often known as an *entropy fix*. More systematic and complex methods that extend the Roe approach whilst avoiding this problem include the *Marquina* solver.

## HLL-type solutions

An alternative type of method simplifies the wave structure even more, by simplifying the number of waves. HLL (for Harten, Lax and van Leer) type solutions assume that

1. there are two waves, both discontinuities, separating a constant central state in the solution, and
2. the waves propagate at the (known) speeds $\xi_{(\pm)}$.

From these assumptions, and the Rankine-Hugoniot conditions, we have the two equations

\begin{align}
  \xi_{(-)} \left[ {\bf q}_m - {\bf q}_l \right] & = {\bf f}_m - {\bf f}_l, \\
  \xi_{(+)} \left[ {\bf q}_r - {\bf q}_m \right] & = {\bf f}_r - {\bf f}_m.
\end{align}

These are immediately solved to give

\begin{align}
  {\bf q}_m & = \frac{\xi_{(+)} {\bf q}_r - \xi_{(-)} {\bf q}_l - {\bf f}_r + {\bf f}_l}{\xi_{(+)} - \xi_{(-)}}, \\
  {\bf f}_m & = \frac{\hat{\xi}_{(+)} {\bf f}_l - \hat{\xi}_{(-)} {\bf f}_r + \hat{\xi}_{(+)} \hat{\xi}_{(-)} \left( {\bf q}_r - {\bf q}_r \right)}{\hat{\xi}_{(+)} - \hat{\xi}_{(-)}},
\end{align}

where

\begin{equation}
  \hat{\xi}_{(-)} = \min(0, \xi_{(-)}), \qquad \hat{\xi}_{(+)} = \max(0, \xi_{(+)}).
\end{equation}

Again it should be noted that, in general, ${\bf f}_m \ne {\bf f}({\bf q}_m)$.

We still need some way to compute the wave speeds $\xi_{(\pm)}$. The simplest method is to make them as large as possible, compatible with stability. This means (via the CFL condition) setting

\begin{equation}
  -\xi_{(-)} = \xi_{(+)} = \frac{\Delta x}{\Delta t}
\end{equation}

which implies that (as the central state is now guaranteed to include the origin, as the waves have different signs)

\begin{equation}
  {\bf f}^* = \frac{1}{2} \left( {\bf f}_l + {\bf f}_r + \frac{\Delta x}{\Delta t} \left[ {\bf q}_l - {\bf q}_r \right] \right).
\end{equation}

This is the *Lax-Friedrichs* flux, as [used in HyperPython](https://github.com/ketch/HyperPython). We can also easily see how the *local* Lax-Friedrichs method, [used in lesson 3 of HyperPython](http://nbviewer.ipython.org/github/ketch/HyperPython/blob/master/Lesson_03_High-resolution_methods.ipynb), comes about: simply choose

\begin{equation}
  -\xi_{(-)} = \xi_{(+)} = \alpha = \min \left( \left| \lambda \left( {\bf q}_l \right) \right|, \left| \lambda \left( {\bf q}_r \right) \right| \right)
\end{equation}

to get

\begin{equation}
  {\bf f}^* = \frac{1}{2} \left( {\bf f}_l + {\bf f}_r + \alpha \left[ {\bf q}_l - {\bf q}_r \right] \right).
\end{equation}

HLL type methods are straightforward to use but typically do not capture linear waves, such as the contact wave in the Euler equations, well. Extending the HLL method by including more waves is possible (see the *HLLC* method in Toro's book as an example), but rapidly increases the complexity of the solver.


```python
from skimage import io
import matplotlib.pyplot as plt
%matplotlib inline

# https://directory.eoportal.org/web/eoportal/satellite-missions/c-missions/copernicus-sentinel-2
url="https://directory.eoportal.org/documents/163813/4091221/Sentinel2_Auto98.jpeg"
image = io.imread(url)
plt.imshow(image)
plt.title("Peruvian mountain scene, 14 July 2017, Sentinel-2\n (credit: ESA, processed by ESA, CC BY-SA 3.0 IGO)")
plt.show()
```

# Using equation with LaTeX notation in a markdown cell

The well known Pythagorean theorem $x^2 + y^2 = z^2$ was
proved to be invalid for other exponents.
Meaning the next equation has no integer solutions:

\begin{equation}
 x^n + y^n = z^n
\end{equation}

```python
import matplotlib
import matplotlib.pyplot as plt
import numpy as np

# Data for plotting
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2 * np.pi * t)

fig, ax = plt.subplots()
ax.plot(t, s)

ax.set(xlabel='time (s)', ylabel='voltage (mV)',
       title='About as simple as it gets, folks')
ax.grid()

fig.savefig("test.png")
plt.show()
```

```python
# !conda list
```

# Introduction

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


```python
%reload_ext autoreload
%autoreload 2
%matplotlib inline

from data_sci.fastai.nlp import *
from sklearn.linear_model import LogisticRegression
```

## IMDB dataset and the sentiment classification task

The [large movie review dataset](http://ai.stanford.edu/~amaas/data/sentiment/) contains a collection of 50,000 reviews from IMDB. The dataset contains an even number of positive and negative reviews. The authors considered only highly polarized reviews. A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10. Neutral reviews are not included in the dataset. The dataset is divided into training and test sets. The training set is the same 25,000 labeled reviews.

The **sentiment classification task** consists of predicting the polarity (positive or negative) of a given text.

To get the dataset, in your terminal run the following commands:

`wget http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz`

`gunzip aclImdb_v1.tar.gz`

`tar -xvf aclImdb_v1.tar`

### Tokenizing and term document matrix creation

```python
# PATH='data/aclImdb/'
PATH='/data/msnow/data_science/imdb/aclImdb/'
names = ['neg','pos']
```

```python
%ls {PATH}
```

    imdbEr.txt  imdb.vocab  README  test/  train/

```python
%ls {PATH}train
```

    labeledBow.feat  pos/    unsupBow.feat  urls_pos.txt
    neg/             unsup/  urls_neg.txt   urls_unsup.txt

```python
%ls {PATH}train/pos | head
```

    0_9.txt
    10000_8.txt
    10001_10.txt
    10002_7.txt
    10003_8.txt
    10004_8.txt
    10005_7.txt
    10006_7.txt
    10007_7.txt
    10008_7.txt
    ls: write error

```python
??texts_labels_from_folders
```

```python
trn,trn_y = texts_labels_from_folders(f'{PATH}train',names)
val,val_y = texts_labels_from_folders(f'{PATH}test',names)
```

Here is the text of the first review

```python
trn[0]
```

    'I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered "controversial" I really had to see this for myself.

The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.

What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.

I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\'t have much of a plot.'

```python
trn_y[0]
```

    0

[`CountVectorizer`](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) converts a collection of text documents to a matrix of token counts (part of `sklearn.feature_extraction.text`).

```python
veczr = CountVectorizer(tokenizer=tokenize)
```

`fit_transform(trn)` finds the vocabulary in the training set. It also transforms the training set into a term-document matrix. Since we have to apply the *same transformation* to your validation set, the second line uses just the method `transform(val)`. `trn_term_doc` and `val_term_doc` are sparse matrices. `trn_term_doc[i]` represents training document i and it contains a count of words for each document for each word in the vocabulary.

```python
trn_term_doc = veczr.fit_transform(trn)
val_term_doc = veczr.transform(val)
```

```python
trn_term_doc
```

    <25000x75132 sparse matrix of type '<class 'numpy.int64'>'
    	with 3749745 stored elements in Compressed Sparse Row format>

```python
trn_term_doc[0]
```

    <1x75132 sparse matrix of type '<class 'numpy.int64'>'
    	with 189 stored elements in Compressed Sparse Row format>

```python
vocab = veczr.get_feature_names(); vocab[5000:5005]
```

    ['aussie', 'aussies', 'austen', 'austeniana', 'austens']

```python
w0 = set([o.lower() for o in trn[0].split(' ')]); w0
```

    {'"controversial"',
     '(no',
     '/>
i',
     '/>the',
     '/>what',
     '1967.',
     '40',
     'a',
     'about',
     'ago,',
     'all',
     'also',
     'am',
     'america.',
     'and',
     'answer',
     'any',
     'anyone',
     'are',
     'arguably',
     'around',
     'artistic',
     'as',
     'asking',
     'at',
     'attentions',
     'average',
     'be',
     'because',
     'being',
     'bergman,',
     'between',
     'between,',
     'boy',
     'but',
     'by',
     'can',
     'centered',
     'certain',
     'cheaply',
     'cinema.',
     'classmates,',
     'commend',
     'considered',
     'controversy',
     'country,',
     'countrymen',
     'curious-yellow',
     'customs',
     'denizens',
     'do',
     'documentary',
     "doesn't",
     'drama',
     'enter',
     'even',
     'ever',
     'everything',
     'fact',
     'fan',
     'far',
     'few',
     'film',
     'filmmakers',
     'films',
     'films.<br',
     'find',
     'first',
     'focus',
     'for',
     'ford,',
     'from',
     'good',
     'had',
     'has',
     'have',
     'heard',
     'her',
     'his',
     'i',
     'if',
     'in',
     'ingmar',
     'intended)',
     'is',
     'issues',
     'it',
     "it's",
     'john',
     'just',
     'kills',
     'learn',
     'lena',
     'life.',
     'like',
     'made',
     'major',
     'make',
     'making',
     'married',
     'me',
     'meat',
     'men.<br',
     'mind',
     'money',
     'much',
     'my',
     'myself.<br',
     'named',
     'not',
     'nudity',
     'of',
     'old',
     'on',
     'opinions',
     'ordinary',
     'particular',
     'people',
     'plot',
     'plot.',
     'political',
     'politicians',
     'politics,',
     'porno.',
     'pornographic',
     'pornographic.',
     'potatoes',
     'pun',
     'purposes',
     'race',
     'rather',
     'reality',
     'really',
     'really,',
     'released',
     'rented',
     'scenes',
     'see',
     'seized',
     'sex',
     'she',
     'shock',
     'shocking,',
     'shot',
     'shown',
     'some',
     'sort',
     'staple',
     'states.',
     'stockholm',
     'store',
     'student',
     'study',
     'such',
     'surrounded',
     'swede',
     'swedish',
     'teacher,',
     'than',
     'that',
     'the',
     'theaters',
     'their',
     'then',
     'therefore',
     'this',
     'thought',
     'to',
     'tried',
     'u.s.',
     'united',
     'video',
     'vietnam',
     'wanting',
     'wants',
     'war',
     'was',
     'what',
     'when',
     'while',
     'who',
     'with',
     'years',
     'young'}

```python
len(w0)
```

    185

```python
veczr.vocabulary_['really']
```

    53936

```python
trn_term_doc[0,53936]
```

    3

```python
trn_term_doc[0,5000]
```

    0

## Naive Bayes

### Theory break

This is to create a markdown style table from a pandas dataframe

```python
tmp = []
tmp.append('This movie was great')
tmp.append('I liked this movie')
tmp.append('This is the worst movie ever')
tmp.append('Bad movie')
veczr_tmp = CountVectorizer(tokenizer=tokenize)
tmp_fit = veczr_tmp.fit_transform(tmp)
df = pd.DataFrame(tmp_fit.toarray(),columns=veczr_tmp.get_feature_names(),index=tmp)

from tabulate import tabulate
print(tabulate(df, headers=df.columns,tablefmt="pipe"))
```

    |                              |   bad |   ever |   great |   i |   is |   liked |   movie |   the |   this |   was |   worst |
    |:-----------------------------|------:|-------:|--------:|----:|-----:|--------:|--------:|------:|-------:|------:|--------:|
    | This movie was great         |     0 |      0 |       1 |   0 |    0 |       0 |       1 |     0 |      1 |     1 |       0 |
    | I liked this movie           |     0 |      0 |       0 |   1 |    0 |       1 |       1 |     0 |      1 |     0 |       0 |
    | This is the worst movie ever |     0 |      1 |       0 |   0 |    1 |       0 |       1 |     1 |      1 |     0 |       1 |
    | Bad movie                    |     1 |      0 |       0 |   0 |    0 |       0 |       1 |     0 |      0 |     0 |       0 |

In general, I want to know, given a specific document (which in our case refers to a review), whether it is a positive or negative review, class 0 or class 1, respectively.  Using Bayes I can determine the probability that a specific document will be in a certain class, e.g., $p\left(c=1\mid d\right)$.

$$ p\left(c=1 \mid d\right) = \dfrac{p\left(d\mid c=1\right) p\left(c=1\right)}{p\left(d\right)} $$

Let's take this one step further before trying to solve, as it will make the math easier.  I don't realy care about the probabilty or a review being positive or negative, I just want to know if it's more likely to be psoitive or negative.  I can extract this information by taking the ratio of the conditional probabilities.

$$\dfrac{p\left(c=1 \mid d\right)}{p\left(c=0 \mid d\right)} $$

If the result is greater than 1, then the review is more likely to belong to class 1, i.e., positive and if the result is less than 1, the review is more likely to be negative, i.e., class 0.

\begin{align}
\dfrac{p\left(c=1 \mid d\right)}{p\left(c=0 \mid d\right)} & = \dfrac{p\left(d\mid c=1\right) p\left(c=1\right)}{p\left(d\right)} \dfrac{p\left(d\right)}{p\left(d\mid c=0\right) p\left(c=0\right)} \\
& = \dfrac{p\left(d\mid c=1\right) p\left(c=1\right)}{p\left(d\mid c=0\right) p\left(c=0\right)} \\
\end{align}

Let's go through each of these terms in the context of the four sample reviews in the following term document matrix.

|                              |   bad |   ever |   great |   i |   is |   liked |   movie |   the |   this |   was |   worst |
|:-----------------------------|------:|-------:|--------:|----:|-----:|--------:|--------:|------:|-------:|------:|--------:|
| This movie was great         |     0 |      0 |       1 |   0 |    0 |       0 |       1 |     0 |      1 |     1 |       0 |
| I liked this movie           |     0 |      0 |       0 |   1 |    0 |       1 |       1 |     0 |      1 |     0 |       0 |
| This is the worst movie ever |     0 |      1 |       0 |   0 |    1 |       0 |       1 |     1 |      1 |     0 |       1 |
| Bad movie                    |     1 |      0 |       0 |   0 |    0 |       0 |       1 |     0 |      0 |     0 |       0 |

$p\left(c=C\right)$ is simply the probability of a document being class 0 or 1.  This is just the number of docuemnts in each class divided by the total number of documents

\begin{align}
p\left(c=0\right) &=2/4 = 0.5 \\
p\left(c=1\right) &= 2/4 = 0.5 \\
\end{align}

$p\left(d\mid c=C\right)$ is the probability of seeing this document given a specific class, $C$.  Since the document is just the words (or in nlp speak, the features) which make it up, we can rewrite these terms as $p\left(f_0,f_1,\ldots,f_p\mid c=0\right)$.  For example, for the first review

$$ p\left(d_0\mid c=0\right) = p\left(f_8, f_6, f_9, f_2\mid c=0\right)$$

Here is where the Naive part of Naive Bayes comes in.  In Naive Bayes we assume that all features are conditionally independent, which means that I can rewrite the previous equation as

$$ p\left(f_8, f_6, f_9, f_2\mid c=C\right) = p\left(f_8 \mid c=C\right) \times p\left(f_6 \mid c=C\right) \times p\left(f_9 \mid c=0\right) \times p\left(f_2 \mid c=C\right) = \prod\limits_{i=8,6,9,2}p\left(f_i \mid c=C\right) $$

Going back to our problem, we can now calculate $p\left(d_0 \mid c=0\right)$ for each feature as the number of times that feature appears in the document divided by the number of times that feature appears in all documents of that class.

$$p\left(f_{this}\mid c=0\right) = 2/2 = 1$$

$$p\left(f_{movie}\mid c=0\right) = 2/2 = 1$$

$$p\left(f_{was}\mid c=0\right) = 1/2 = 0.5$$

$$p\left(f_{great}\mid c=0\right) = 1/2 = 0.5$$

$$p\left(d_0 \mid c=0\right) = 1\times 1 \times 0.5 \times 0.5 = 0.25 $$

If we try and repeat the same procedure for the other class, we end up with a problem.  What happens if that feature never appears in that class.

$$p\left(f_{great}\mid c=1\right) = 1/0 = ???$$

To get around this problem we add an additional row to our term document matrix which contains a 1 in every entry.  Intuitively this row represents the idea that there is never a zero percent chance of some word appearing.  It might be infinitesimal, but it is greater than zero. This row of ones is used just for calculating $p\left(d \mid c=C\right)$

|                              |   bad |   ever |   great |   i |   is |   liked |   movie |   the |   this |   was |   worst |
|:-----------------------------|------:|-------:|--------:|----:|-----:|--------:|--------:|------:|-------:|------:|--------:|
| This movie was great         |     0 |      0 |       1 |   0 |    0 |       0 |       1 |     0 |      1 |     1 |       0 |
| I liked this movie           |     0 |      0 |       0 |   1 |    0 |       1 |       1 |     0 |      1 |     0 |       0 |
| This is the worst movie ever |     0 |      1 |       0 |   0 |    1 |       0 |       1 |     1 |      1 |     0 |       1 |
| Bad movie                    |     1 |      0 |       0 |   0 |    0 |       0 |       1 |     0 |      0 |     0 |       0 |
| **ones**                     |     1 |      1 |       1 |   1 |    1 |       1 |       1 |     1 |      1 |     1 |       1 |

We can now recalculate the probabilities:

$$p\left(f_{this}\mid c=0\right) = (2+1)/3 = 1$$

$$p\left(f_{movie}\mid c=0\right) = (2+1)/3 = 1$$

$$p\left(f_{was}\mid c=0\right) = (1+1)/3 = 0.667$$

$$p\left(f_{great}\mid c=0\right) = (1+1)/3 = 0.667$$

$$p\left(d_0 \mid c=0\right) = 1\times 1 \times 0.667 \times 0.667 = 0.444 $$

Repeat for the other class

$$p\left(f_{this}\mid c=1\right) = (1+1)/3 = 0.667$$

$$p\left(f_{movie}\mid c=1\right) = (2+1)/3 = 1$$

$$p\left(f_{was}\mid c=1\right) = (0+1)/3 = 0.333$$

$$p\left(f_{great}\mid c=1\right) = (0+1)/3 = 0.333$$

$$p\left(d_0 \mid c=1\right) = 0.667 \times 1 \times 0.333 \times 0.333 = 0.074 $$

Now to answer our original question we just need to take the ratios of these two probabilities

$$ \dfrac{p\left(d_0 \mid c=0\right)}{p\left(d_0 \mid c=1\right)} = \dfrac{0.444}{0.074} = 6$$

This tells us that it is 6 times more likely that review 1 belongs to class 0, than class 1.

As an aside: If we didn't use Naive Bayes the equation would be much harder to solve as the $p\left(f_8, f_6, f_9, f_2\mid c=0\right)$ would expand into a much harder term to solve:

$$ p\left(f_8, f_6, f_9, f_2\mid c=0\right) = p\left(f_8 \mid c=0\right) \times p\left(f_6\mid f_8, c=0\right) \times p\left(f_9, f_2\mid f_8, f_6, c=0\right) \times p\left(f_2\mid f_8, f_6, f_9, c=0\right) $$

### Theory break over

As stated above, the probability of each feature appearing in a document of a specific class is just the ratio of the number of times that feature appears in that class (plus 1) to the number of documents in that class (plus 1).  The ratio of ratios is then half of the equation we need to solve the problem.  To make the multiplication and division less likely to go to zero or infinity we can also convert everything to logs

```python
def pr(y_i):
    p = x[y==y_i].sum(0)
    return (p+1) / ((y==y_i).sum()+1)

x=trn_term_doc
y=trn_y

r = np.log(pr(1)/pr(0))
b = np.log((y==1).mean() / (y==0).mean())
```

Here is the formula for Naive Bayes.

Instead of calculating the probabilities for each document individiually we can just use matrix multiplication.

```python
pre_preds = val_term_doc @ r.T + b
preds = pre_preds.T>0
(preds==val_y).mean()
```

    0.81655999999999995

...and binarized Naive Bayes (where I don't care how often I've seen it, just if I have seen it or not)

```python
x=trn_term_doc.sign()
r = np.log(pr(1)/pr(0))

pre_preds = val_term_doc.sign() @ r.T + b
preds = pre_preds.T>0
(preds==val_y).mean()
```

    0.83016000000000001

## Logistic regression

Here is how we can fit logistic regression where the features are the unigrams.

```python
m = LogisticRegression(C=1e8, dual=True)
m.fit(x, y)
preds = m.predict(val_term_doc)
(preds==val_y).mean()
```

    0.83328000000000002

...and the binarized version

```python
m = LogisticRegression(C=1e8, dual=True)
m.fit(trn_term_doc.sign(), y)
preds = m.predict(val_term_doc.sign())
(preds==val_y).mean()
```

    0.85519999999999996

...and the regularized version (The C paramater, the closer to 1, the greater the regularization)

```python
m = LogisticRegression(C=0.1, dual=True)
m.fit(x, y)
preds = m.predict(val_term_doc)
(preds==val_y).mean()
```

    0.84872000000000003

...and the regularized binarized version

```python
m = LogisticRegression(C=0.1, dual=True)
m.fit(trn_term_doc.sign(), y)
preds = m.predict(val_term_doc.sign())
(preds==val_y).mean()
```

    0.88404000000000005

### Trigram with NB features

Our next model is a version of logistic regression with Naive Bayes features described [here](https://www.aclweb.org/anthology/P12-2018). For every document we compute binarized features as described above, but this time we use bigrams and trigrams too. Each feature is a log-count ratio. A logistic regression model is then trained to predict sentiment.

```python
veczr =  CountVectorizer(ngram_range=(1,3), tokenizer=tokenize, max_features=800000)
trn_term_doc = veczr.fit_transform(trn)
val_term_doc = veczr.transform(val)
```

```python
trn_term_doc.shape
```

    (25000, 800000)

```python
vocab = veczr.get_feature_names()
```

```python
vocab[200000:200005]
```

    ['by vast', 'by vengeance', 'by vengeance .', 'by vera', 'by vera miles']

```python
y=trn_y
x=trn_term_doc.sign()
val_x = val_term_doc.sign()
```

```python
r = np.log(pr(1) / pr(0))
b = np.log((y==1).mean() / (y==0).mean())
```

Here we fit regularized logistic regression where the features are the trigrams.

```python
m = LogisticRegression(C=0.1, dual=True)
m.fit(x, y);

preds = m.predict(val_x)
(preds.T==val_y).mean()
```

    0.90500000000000003

Here is the $\text{log-count ratio}$ `r`.

```python
r.shape, r
```

    ((1, 800000), matrix([[-0.05468386, -0.16100472, -0.24783616, ...,  1.09861229,
              -0.69314718, -0.69314718]]))

```python
np.exp(r)
```

    matrix([[ 0.94678442,  0.85128806,  0.7804878 , ...,  3.        ,
              0.5       ,  0.5       ]])

Here we fit regularized logistic regression where the features are the trigrams' log-count ratios.

This is not equivalent to just multiplying the weight by the ratios as the weights get regularized while the input values do not.  Thus when you multiply the input values by the naive Bayes ratios you are essentially saying that you beleive the ratios and that the model should not alter them unless it has a good reason to.

```python
x_nb = x.multiply(r)
m = LogisticRegression(dual=True, C=0.1)
m.fit(x_nb, y);

val_x_nb = val_x.multiply(r)
preds = m.predict(val_x_nb)
(preds.T==val_y).mean()
```

    0.91768000000000005

## fastai NBSVM++

```python
sl=2000
```

```python
# Here is how we get a model from a bag of words
md = TextClassifierData.from_bow(trn_term_doc, trn_y, val_term_doc, val_y, sl)
```

```python
learner = md.dotprod_nb_learner()
learner.fit(0.02, 1, wds=1e-6, cycle_len=1)
```

    A Jupyter Widget

    [ 0.       0.0251   0.12003  0.91552]

```python
learner.fit(0.02, 2, wds=1e-6, cycle_len=1)
```

    A Jupyter Widget

    [ 0.       0.02014  0.11387  0.92012]
    [ 1.       0.01275  0.11149  0.92124]

```python
learner.fit(0.02, 2, wds=1e-6, cycle_len=1)
```

    A Jupyter Widget

    [ 0.       0.01681  0.11089  0.92129]
    [ 1.       0.00949  0.10951  0.92223]

## References

* Baselines and Bigrams: Simple, Good Sentiment and Topic Classification. Sida Wang and Christopher D. Manning [pdf](https://www.aclweb.org/anthology/P12-2018)

```python

```