# ORION Orientation Estimation Using Commodity Wi-Fi

With MIMO, Wi-Fi led the way to the adoption of antenna array signal processing techniques for fine-grained localization using commodity hardware. MIMO techniques, previously exclusive to specific domains of applications for instance radar systems, allow to consider estimating the orientation in space of a device equipped with COTS Wi-Fi chip. In fact, the availability channel state information measurement in recent Wi-Fi chips, makes it a candidate readily available for experimentation. Accordingly, we propose the ORION system to estimate the orientation (heading and yaw) of a MIMO Wi-Fi equipped object, relying on a joint estimation of the angle of arrival and the angle of departure.

## Implementation

### Experimentation Setup
We have implemented the system using the Intel Wi-Fi Link 5300 NICs. Indeed, we have used the 533AN_MMW (Full) model with the following SPS references: 480986-0001 0E, 506679-001 0B, 480986-0010A. The firmware of this COTS Wi-Fi card was modified in order to extract the CSI matrices through the Intel CSI tool for 802.11n HT packets. In case of OFDM systems, as for instance Wi-Fi, we are able to extract a CSI matrix for each sub-carrier. In our case the Wireless NIC offers up to 30 subcarriers. We use the 5 GHz band with 20MHz of bandwidth. We set up our wireless cards in the injection mode, which avoids the need of association with an AP and allows raw Wi-Fi packets transmission.

We setup two uniform linear antenna arrays (ULAs). We use a classical 5dBi omnidirectional compatible with 2.4 GHz as well as the 5GHz band. As we are using the 5.32 GHz band for our system, we respect a 2.8cm of inter-antenna spacing, which corresponds to half the wavelength. These ULAs are connected to the terminal through extension cables in order to have liberty of movement, which is essential for our experimentation. The cables are 3 meters long and compatible with both the Wi-Fi bands with 2dB signal attenuation for 2.4 GHz band.

###  Wireless Card Configuration
The transmission is controlled by disabling the antenna selection algorithm and by specifying the desired number of antennas, streams and transmission technique which is in our case the Spatial Multiplexing (SM). The rotation of the antenna array is performed using extension cables that connect the coplanar transmitter and receiver antenna arrays to the access points.
In order to specify correctly the antenna mapping as well the number of streams used for the transmission, we change the monitor_tx_rate file, after making sure that all the previous steps for collecting CSI were respected, we could for example launch the following command:

```bash
sudo echo 0x4101 |sudo tee /sys/kernel/debug/ieee80211/phy0/iwlwifi/iwldvm/debug/monitor_tx_rate
```
This command  is meant for sending 1 stream on the first antenna.
In fact there the number of streams that are sent is dependent on the MCS index used .

If we want to build a rate index we need to choose and appropriate MCS index corresponding to the number of streams. For example, for sending two streams we choose the MCS index number 9 (make sure that the index is in hexadecimal). For choosing the antenna involved in the transmission we need to apply the appropriate mask:
```
0x04000 for the first antenna
0x08000 for the second antenna
0x10000 for the third antenna
```
We use the HT leagacy packet format by using the flag 0x100.

And thus if we want to send 2 streams on the second and third antenna using HT leagacy packets, the appropriate rate would be:
```
0x18109=0x08000+0x10000+0x100+0x9
```
And thus the command used is as follows:
```bash
sudo echo 0x18109 |sudo tee /sys/kernel/debug/ieee80211/phy0/iwlwifi/iwldvm/debug/monitor_tx_rate.

```

For further details check: https://github.com/dhalperi/linux-80211n-csitool/blob/csitool-3.13/drivers/net/wireless/iwlwifi/dvm/commands.h#L245-L334

For the transmission we choose the injection mode based on lorcon-old. Lorcon, which stands for Loss Of Radio CONnectivity is library for injecting 802.11 frames https://github.com/dhalperi/lorcon-old.

If you choose to use nepi-ng for running your experiment, you should apply an appropriate rate index in https://github.com/parmentelat/r2lab/blob/public/demos/jobs-angle-measure/angle-measure.sh#L128-L134

## Joint Angle of arrival Angle of Departure Estimation

We  jointly estimate AoA and AoD using a 2D MUSIC algorithm on phase corrected CSI matrices collected through out the experiment for a subcarrier.

The figure shown below represents the values computed for the following pseudo-spectrum:

$$
\begin{equation}
\textbf{P}(\theta_i,\phi_i)=\frac{1}{(\textbf{a}_r(\theta_i) \otimes \textbf{a}_t(\phi_i))^H.\textbf{Q}_n.\textbf{Q}_n^H(\textbf{a}_r(\theta_i) \otimes \textbf{a}_t(\phi_i))}
\end{equation}
$$
with $\theta_i $ is the angle of arrival and $\phi_i$ is the angle of departure. The peak represents the AoA and AoD estimates.

```python
import matplotlib.pyplot as plt
%matplotlib inline
from music import *
csi_corr='./log/csi_corr.dat'
csi_target='./log/csi_target.dat'
music(csi_corr, csi_target, 2, 3, 0.5, 0.5,640)
#music_pl(s, 2, 3, 0.5, 0.5,1)
```

```python
%matplotlib inline
from music_pl import *
from IPython import display
import time

x = []
y = []
z = []

s = phase_correction(csi_corr, csi_target)

for i in range(1900):
    x = np.append(x, i)
    o=music_pl(s, 2, 3, 0.5, 0.5,i)
    y = np.append(y, o[0])
    z = np.append(z, o[1])
    plt.gca().cla()
    plt.plot(x,y,label='DOA')
    plt.plot(x,z,label='DOD')
    plt.ylabel('Angle in degrees')
    plt.xlabel('packet number')
    plt.legend()
    plt.ylim(-90, 90)
    display.clear_output(wait=True)
    display.display(plt.gcf())
```

```python
angle=np.array(np.squeeze([[y],[z]]))
angle.shape
```

    (2, 1942)

```python
import pandas as pd
df = pd.DataFrame(angle)
df.to_csv("exp_meas3.csv",header=None)
```

###  Smoothing Angle Estimations

As shown in the figure above, the estimates are jittery with a strong presence of outliers. Thus, in order to smoothen our estimates, we apply an outlier detector on the measurement data before using a Kalman filter to mitigate the statistical noise.

```python
plt.figure()
plt.grid(True)
plt.ylabel('Angle in degrees')
plt.xlabel('packet number')
plt.plot(y,'b',linewidth='1',label='DOA')
plt.plot(z,'r',linewidth='1',label='DOD')
plt.tick_params(labelsize=14)
plt.legend()
plt.ylim(-90, 90)
plt.show()
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
from kalman_fil import *

DOA, DOD,angles1,angles2=kalman_fil("exp_meas.csv")
plt.figure(figsize=(7,7))
plt.grid(True)
plt.ylabel('Angle in degrees')
plt.xlabel('packet number')

plt.subplot(2, 1, 2)
plt.plot(DOA,'b',linewidth='2',label='DOA')
plt.plot(DOD,'r',linewidth='2',label='DOD')
plt.tick_params(labelsize=14)
plt.legend()
plt.grid(True)
plt.ylabel('Angle in degrees')
plt.xlabel('packet number')
plt.ylim(-60, 60)
plt.subplot(2, 1, 1)
plt.plot(angles1,'b',linewidth='2',label='DOA')
plt.plot(angles2,'r',linewidth='2',label='DOD')
plt.grid(True)
plt.ylabel('Angle in degrees')
plt.xlabel('packet number')
plt.tick_params(labelsize=14)
plt.legend()
plt.ylim(-60, 60)
plt.show()
```

So after applying an outlier removal using a Hampel identifier and a Kalman filter on our estimated data we have significantly decreased the noise and increased the overall accuracy.


```python
! pip install qiskit
```

    Collecting qiskit
      Downloading qiskit-0.34.0.tar.gz (13 kB)
    Collecting qiskit-terra==0.19.1
      Downloading qiskit_terra-0.19.1-cp37-cp37m-manylinux2010_x86_64.whl (6.4 MB)
         |████████████████████████████████| 6.4 MB 4.2 MB/s
    Collecting qiskit-aer==0.10.1
      Downloading qiskit_aer-0.10.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (18.0 MB)
         |████████████████████████████████| 18.0 MB 50.8 MB/s
    Collecting qiskit-ibmq-provider==0.18.3
      Downloading qiskit_ibmq_provider-0.18.3-py3-none-any.whl (238 kB)
         |████████████████████████████████| 238 kB 56.3 MB/s
    Collecting qiskit-ignis==0.7.0
      Downloading qiskit_ignis-0.7.0-py3-none-any.whl (200 kB)
         |████████████████████████████████| 200 kB 65.7 MB/s
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.1->qiskit) (1.4.1)
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.1->qiskit) (1.19.5)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.24.3)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.8.2)
    Collecting requests-ntlm>=1.1.0
      Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.23.0)
    Collecting websocket-client>=1.0.1
      Downloading websocket_client-1.2.3-py3-none-any.whl (53 kB)
         |████████████████████████████████| 53 kB 2.2 MB/s
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.10.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)
         |████████████████████████████████| 1.4 MB 66.2 MB/s
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.1->qiskit) (5.4.8)
    Collecting ply>=3.10
      Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 5.3 MB/s
    Collecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (943 kB)
         |████████████████████████████████| 943 kB 54.5 MB/s
    Collecting python-constraint>=1.4
      Downloading python-constraint-1.4.0.tar.bz2 (18 kB)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.1->qiskit) (0.3.4)
    Collecting stevedore>=3.0.0
      Downloading stevedore-3.5.0-py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.0 MB/s
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.1->qiskit) (1.7.1)
    Collecting symengine>=0.8
      Downloading symengine-0.8.1-cp37-cp37m-manylinux2010_x86_64.whl (38.2 MB)
         |████████████████████████████████| 38.2 MB 1.2 MB/s
    Collecting scipy>=1.0
      Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
         |████████████████████████████████| 38.1 MB 1.2 MB/s
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2.10)
    Collecting ntlm-auth>=1.0.2
      Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Collecting cryptography>=1.3
      Downloading cryptography-36.0.1-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
         |████████████████████████████████| 3.6 MB 67.7 MB/s
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (2.21)
    Collecting pbr!=2.1.0,>=2.0.0
      Downloading pbr-5.8.0-py2.py3-none-any.whl (112 kB)
         |████████████████████████████████| 112 kB 58.4 MB/s
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.1->qiskit) (4.8.2)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.1->qiskit) (3.6.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.1->qiskit) (3.10.0.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.19.1->qiskit) (1.2.1)
    Building wheels for collected packages: qiskit, python-constraint
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.34.0-py3-none-any.whl size=11764 sha256=9f1780476d5d028b08ccbe371f7d4884784473b48d20f51bef7764581c9fc058
      Stored in directory: /root/.cache/pip/wheels/be/f4/26/53ffcc0c3a1303116016eaf20c4f0c8dc125bc3b3c7afa2347
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24081 sha256=97fa637431d714866ec2d61b3218f27812fd62dc1569c5a918dc4537d4567d47
      Stored in directory: /root/.cache/pip/wheels/07/27/db/1222c80eb1e431f3d2199c12569cb1cac60f562a451fe30479
    Successfully built qiskit python-constraint
    Installing collected packages: pbr, tweedledum, symengine, stevedore, scipy, retworkx, python-constraint, ply, ntlm-auth, cryptography, websocket-client, requests-ntlm, qiskit-terra, qiskit-ignis, qiskit-ibmq-provider, qiskit-aer, qiskit
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed cryptography-36.0.1 ntlm-auth-1.5.0 pbr-5.8.0 ply-3.11 python-constraint-1.4.0 qiskit-0.34.0 qiskit-aer-0.10.1 qiskit-ibmq-provider-0.18.3 qiskit-ignis-0.7.0 qiskit-terra-0.19.1 requests-ntlm-1.1.0 retworkx-0.10.2 scipy-1.7.3 stevedore-3.5.0 symengine-0.8.1 tweedledum-1.1.1 websocket-client-1.2.3

```python
import qiskit
import PIL
import pandas as pd
import numpy as np
import os
import time
import operator
import tensorflow as tf
import matplotlib.pyplot as plt
import math

from qiskit import *
from qiskit.circuit import Reset
from qiskit.test.mock import FakeMontreal
from qiskit.compiler import transpile
from PIL import Image
from time import perf_counter
from tensorflow.keras import layers, models
from math import *
```

```python
def construct_data_frame(dir):
  nodule_types = ['NORMAL', 'BENIGN', 'MALIGNANT']
  data = []
  for nodule_num, nodule_type in enumerate(nodule_types):
    i = 1
    for image in os.listdir(os.path.join(dir, nodule_type)):
      image_file = Image.open('{}/{}/{}'.format(dir, nodule_type, image))
      data.append(['{}/{}/{}'.format(dir, nodule_type, image), nodule_num, nodule_type])
      if i == 20:
        break
      i += 1
  table = pd.DataFrame(data, columns = ['File', 'NoduleNum', 'NoduleTypes'])
  table = table.sample(frac=1).reset_index(drop=True)
  return table
```

```python
def quantum_filter(input_vals):
  qreg = QuantumRegister(5)
  creg = ClassicalRegister(1)
  circ = QuantumCircuit(qreg, creg)
  for qubit in qreg:
   circ.h(qubit)
  for i in range(4):
    circ.ry(pi*input_vals[i], i)
  circ.z(qreg[0])
  circ.h(qreg[3])
  circ.cx(qreg[2], qreg[3])
  circ.h(qreg[2])
  circ.h(qreg[3])
  circ.cx(qreg[1], qreg[2])
  circ.h(qreg[3])
  circ.h(qreg[1])
  circ.h(qreg[2])
  circ.cx(qreg[0], qreg[3])
  circ.h(qreg[2])
  circ.h(qreg[3])
  circ.cx(qreg[0], qreg[2])
  circ.h(qreg[3])
  circ.cx(qreg[0], qreg[1])
  circ.h(qreg[2])
  circ.h(qreg[1])
  circ.ccx(qreg[1], qreg[2], qreg[3])
  circ.h(qreg[2])
  circ.h(qreg[3])
  circ.ccx(qreg[0], qreg[1], qreg[2])
  circ.h(qreg[3])
  circ.h(qreg[2])
  circ.mcx([qreg[0], qreg[1], qreg[2]], qreg[3])
  circ.z(qreg[2])
  circ.h(qreg[3])
  circ.z(qreg[3])
  circ.h(qreg[3])
  circ.cx(qreg[2], qreg[3])
  circ.h(qreg[2])
  circ.h(qreg[3])
  circ.h(qreg[3])
  circ.cx(qreg[1], qreg[3])
  circ.h(qreg[1])
  circ.h(qreg[3])
  circ.h(qreg[3])
  circ.cx(qreg[0], qreg[3])
  circ.cx(qreg[0], qreg[2])
  circ.h(qreg[3])
  circ.cx(qreg[0], qreg[1])
  circ.h(qreg[2])
  circ.h(qreg[3])
  circ.x(qreg[0])
  circ.ccx(qreg[1], qreg[2], qreg[3])
  circ.h(qreg[1])
  circ.h(qreg[2])
  circ.h(qreg[3])
  circ.x(qreg[1])
  circ.x(qreg[2])
  circ.h(qreg[3])
  circ.x(qreg[3])
  circ.mcx([qreg[0], qreg[1], qreg[2], qreg[3]], qreg[4])
  circ.measure(qreg[4], creg)
  counts = execute(circ, Aer.get_backend('aer_simulator'), shots = 1000).result().get_counts(circ)
  key = max(counts.items(), key=operator.itemgetter(1))[0]
  return key == '1'
```

```python
def quantum_preprocess(train_nparrays):
  cnn_input = [] # input into cnn after quanvolution (preprocessing step)
  for curr_image in train_nparrays:
    dimensions = curr_image.shape
    num_rows_original = dimensions[0]
    num_cols_original = dimensions[1]
    modified_image_nparray = np.zeros((num_rows_original-1, num_cols_original-1))
    for row in range(num_rows_original-1):
      for col in range(num_cols_original-1):
        filter_input = [curr_image[row][col],
                        curr_image[row][col+1],
                        curr_image[row+1][col],
                        curr_image[row+1][col+1]]
        filter_output = quantum_filter(filter_input)
        if filter_output:
          modified_image_nparray[row][col] = 1 #check poss vals
    cnn_input.append(modified_image_nparray)
  return cnn_input
```

```python
# load data and preprocess
train_data = construct_data_frame('/content/drive/MyDrive/QC-Final-Project/Modified_Dataset/Train')
test_data = construct_data_frame('/content/drive/MyDrive/QC-Final-Project/Modified_Dataset/Test')

train_nparrays = []
train_nodule_types = []
for col, row in train_data.iterrows():
  image = Image.open(row['File'])
  image = image.resize((10, 10))
  image.save(row['File'])
  train_nparrays.append(np.asarray(Image.open(row['File'])))
  train_nodule_types.append(row['NoduleNum'])

test_nparrays = []
test_nodule_types = []
for col, row in test_data.iloc[0:20].iterrows():
  image = Image.open(row['File'])
  image = image.resize((10, 10))
  image.save(row['File'])
  test_nparrays.append(np.asarray(Image.open(row['File'])))
  test_nodule_types.append(test_data['NoduleNum'])
```

```python
cnn_input = np.array(quantum_preprocess(train_nparrays))
train_nodule_types = np.array(train_nodule_types)
test_nparrays = np.array(quantum_preprocess(test_nparrays))
test_nodule_types = np.array(test_nodule_types)
```

```python
# train
qcnn = models.Sequential()
qcnn.add(layers.Conv2D(32, (2, 2), activation = 'relu', input_shape=(9, 9, 1)))
qcnn.add(layers.MaxPooling2D(3, 3))
qcnn.add(layers.Conv2D(32, (1, 1), activation = 'relu', input_shape=(None, 2, 2, 10)))
qcnn.add(layers.Flatten())
qcnn.add(layers.Dense(3, activation='softmax'))
qcnn.compile(optimizer = 'adam',
             loss = "sparse_categorical_crossentropy",
             metrics = ["accuracy"])
history = qcnn.fit(cnn_input,
                   train_nodule_types,
                   validation_split = 0.2,
                   epochs = 10,
                   verbose = 2)
```

    Epoch 1/10
    2/2 - 1s - loss: 1.1290 - accuracy: 0.3125 - val_loss: 1.0477 - val_accuracy: 0.4167 - 565ms/epoch - 282ms/step
    Epoch 2/10
    2/2 - 0s - loss: 1.1155 - accuracy: 0.3125 - val_loss: 1.0742 - val_accuracy: 0.3333 - 30ms/epoch - 15ms/step
    Epoch 3/10
    2/2 - 0s - loss: 1.1058 - accuracy: 0.2500 - val_loss: 1.0998 - val_accuracy: 0.2500 - 24ms/epoch - 12ms/step
    Epoch 4/10
    2/2 - 0s - loss: 1.0974 - accuracy: 0.2917 - val_loss: 1.1250 - val_accuracy: 0.1667 - 23ms/epoch - 11ms/step
    Epoch 5/10
    2/2 - 0s - loss: 1.0914 - accuracy: 0.4167 - val_loss: 1.1514 - val_accuracy: 0.0000e+00 - 25ms/epoch - 13ms/step
    Epoch 6/10
    2/2 - 0s - loss: 1.0863 - accuracy: 0.4167 - val_loss: 1.1771 - val_accuracy: 0.0000e+00 - 23ms/epoch - 12ms/step
    Epoch 7/10
    2/2 - 0s - loss: 1.0815 - accuracy: 0.4167 - val_loss: 1.2017 - val_accuracy: 0.0000e+00 - 25ms/epoch - 13ms/step
    Epoch 8/10
    2/2 - 0s - loss: 1.0773 - accuracy: 0.4167 - val_loss: 1.2235 - val_accuracy: 0.0000e+00 - 38ms/epoch - 19ms/step
    Epoch 9/10
    2/2 - 0s - loss: 1.0791 - accuracy: 0.4167 - val_loss: 1.2462 - val_accuracy: 0.0000e+00 - 42ms/epoch - 21ms/step
    Epoch 10/10
    2/2 - 0s - loss: 1.0752 - accuracy: 0.4167 - val_loss: 1.2616 - val_accuracy: 0.0000e+00 - 28ms/epoch - 14ms/step

```python
# accuracy and loss plots
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
```