```python
!pip install qiskit
```

    Collecting qiskit
      Downloading https://files.pythonhosted.org/packages/6f/61/cb7506e17a2566dc8a31a3e1924d91ac0bdd8ff07c71ec698c06647b6306/qiskit-0.26.2.tar.gz
    Collecting qiskit-terra==0.17.4
      Downloading https://files.pythonhosted.org/packages/b3/0c/3c7a8dd451dae0907263e9de9e3e34909e15e18c88a589b44581972c8511/qiskit_terra-0.17.4-cp37-cp37m-manylinux2010_x86_64.whl (6.0MB)
         |████████████████████████████████| 6.0MB 14.5MB/s
    Collecting qiskit-aer==0.8.2
      Downloading https://files.pythonhosted.org/packages/c2/d2/6ff15c370b5465b32529b528bf3f4ce1e01f74498be16203aa1c04b67022/qiskit_aer-0.8.2-cp37-cp37m-manylinux2010_x86_64.whl (18.0MB)
         |████████████████████████████████| 18.0MB 231kB/s
    Collecting qiskit-ibmq-provider==0.13.1
      Downloading https://files.pythonhosted.org/packages/be/99/74bbb901f88603a7d850d4889abc06d81ba702e4227151f4a5b66f2631fe/qiskit_ibmq_provider-0.13.1-py3-none-any.whl (228kB)
         |████████████████████████████████| 235kB 59.0MB/s
    Collecting qiskit-ignis==0.6.0
      Downloading https://files.pythonhosted.org/packages/54/be/a13c828e457e09d979667a61bddbd8c7246aafa94e2501b6a9154429cbea/qiskit_ignis-0.6.0-py3-none-any.whl (207kB)
         |████████████████████████████████| 215kB 53.3MB/s
    Collecting qiskit-aqua==0.9.1
      Downloading https://files.pythonhosted.org/packages/88/79/392c57b978decbb24b902344b536af52c40a751aed0ebbaefa8bc2964cb5/qiskit_aqua-0.9.1-py3-none-any.whl (2.1MB)
         |████████████████████████████████| 2.1MB 38.1MB/s
    Collecting retworkx>=0.8.0
      Downloading https://files.pythonhosted.org/packages/9e/cd/70d436f170aa1ead2ac9e4c19c8838633355d48b530f09455eab0af2f98e/retworkx-0.9.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4MB)
         |████████████████████████████████| 1.5MB 33.3MB/s
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.17.4->qiskit) (1.7.1)
    Collecting ply>=3.10
      Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
         |████████████████████████████████| 51kB 5.6MB/s
    Requirement already satisfied: scipy>=1.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.17.4->qiskit) (1.4.1)
    Collecting python-constraint>=1.4
      Downloading https://files.pythonhosted.org/packages/37/8b/5f1bc2734ca611943e1d6733ee244238679f6410a10cd45ede55a61a8402/python-constraint-1.4.0.tar.bz2
    Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.17.4->qiskit) (1.19.5)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.17.4->qiskit) (5.4.8)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.17.4->qiskit) (2.8.1)
    Collecting fastjsonschema>=2.10
      Downloading https://files.pythonhosted.org/packages/d1/fb/ea090e917b18320f79be31d754bbe496b715175e865603cfce1eaed2e774/fastjsonschema-2.15.1-py3-none-any.whl
    Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.17.4->qiskit) (2.6.0)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.17.4->qiskit) (0.3.3)
    Collecting pybind11>=2.6
      Downloading https://files.pythonhosted.org/packages/8d/43/7339dbabbc2793718d59703aace4166f53c29ee1c202f6ff5bf8a26c4d91/pybind11-2.6.2-py2.py3-none-any.whl (191kB)
         |████████████████████████████████| 194kB 44.0MB/s
    Requirement already satisfied: nest-asyncio!=1.1.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.13.1->qiskit) (1.5.1)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.13.1->qiskit) (1.24.3)
    Collecting websockets>=8
      Downloading https://files.pythonhosted.org/packages/84/64/78c2b3fe37730b30dca3c93d1f7f4a4286767f86e7c04cf3571b39bc2fb7/websockets-9.1-cp37-cp37m-manylinux2010_x86_64.whl (103kB)
         |████████████████████████████████| 112kB 47.3MB/s
    Collecting requests-ntlm>=1.1.0
      Downloading https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.13.1->qiskit) (2.23.0)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.6.0->qiskit) (56.1.0)
    Collecting quandl<=3.6.0
      Downloading https://files.pythonhosted.org/packages/c2/58/9f0e69d836045e3865d263e9ed49f42b23a58526fdabb30f74c430baee3f/Quandl-3.6.0-py2.py3-none-any.whl
    Requirement already satisfied: fastdtw<=0.3.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.1->qiskit) (0.3.4)
    Requirement already satisfied: pandas<=1.2.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.1->qiskit) (1.1.5)
    Collecting docplex<=2.20.204; sys_platform != "darwin"
      Downloading https://files.pythonhosted.org/packages/87/99/6f7c219b39fd58c84688ad0713eb932bfcf6be81fc74519e43ea9c915b56/docplex-2.20.204.tar.gz (611kB)
         |████████████████████████████████| 614kB 35.8MB/s
    Collecting dlx<=1.0.4
      Downloading https://files.pythonhosted.org/packages/54/c0/b8fb5bb727e983b6f5251433ef941b48f38c65bb0bd6ec509e9185bcd406/dlx-1.0.4.tar.gz
    Requirement already satisfied: h5py<=3.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.1->qiskit) (3.1.0)
    Requirement already satisfied: scikit-learn<=0.24.1,>=0.20.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.1->qiskit) (0.22.2.post1)
    Collecting yfinance<=0.1.55
      Downloading https://files.pythonhosted.org/packages/7a/e8/b9d7104d3a4bf39924799067592d9e59119fcfc900a425a12e80a3123ec8/yfinance-0.1.55.tar.gz
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.17.4->qiskit) (1.2.1)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-terra==0.17.4->qiskit) (1.15.0)
    Collecting ntlm-auth>=1.0.2
      Downloading https://files.pythonhosted.org/packages/ff/84/97c550164b54942b0e908c31ef09d9469f3ba4cd7332a671e2125732f63b/ntlm_auth-1.5.0-py2.py3-none-any.whl
    Collecting cryptography>=1.3
      Downloading https://files.pythonhosted.org/packages/b2/26/7af637e6a7e87258b963f1731c5982fb31cd507f0d90d91836e446955d02/cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2MB)
         |████████████████████████████████| 3.2MB 36.2MB/s
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.13.1->qiskit) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.13.1->qiskit) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.13.1->qiskit) (2020.12.5)
    Collecting inflection>=0.3.1
      Downloading https://files.pythonhosted.org/packages/59/91/aa6bde563e0085a02a435aa99b49ef75b0a4b062635e606dab23ce18d720/inflection-0.5.1-py2.py3-none-any.whl
    Requirement already satisfied: more-itertools in /usr/local/lib/python3.7/dist-packages (from quandl<=3.6.0->qiskit-aqua==0.9.1->qiskit) (8.7.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas<=1.2.3->qiskit-aqua==0.9.1->qiskit) (2018.9)
    Requirement already satisfied: cached-property; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from h5py<=3.1.0->qiskit-aqua==0.9.1->qiskit) (1.5.2)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn<=0.24.1,>=0.20.0->qiskit-aqua==0.9.1->qiskit) (1.0.1)
    Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from yfinance<=0.1.55->qiskit-aqua==0.9.1->qiskit) (0.0.9)
    Collecting lxml>=4.5.1
      Downloading https://files.pythonhosted.org/packages/30/c0/d0526314971fc661b083ab135747dc68446a3022686da8c16d25fcf6ef07/lxml-4.6.3-cp37-cp37m-manylinux2014_x86_64.whl (6.3MB)
         |████████████████████████████████| 6.3MB 32.4MB/s
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.13.1->qiskit) (1.14.5)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.13.1->qiskit) (2.20)
    Building wheels for collected packages: qiskit, python-constraint, docplex, dlx, yfinance
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.26.2-cp37-none-any.whl size=10491 sha256=938a68eae297c0d3ed016ab50ed6996f7c73c766851b97a48900c6dd0a20a90c
      Stored in directory: /root/.cache/pip/wheels/89/89/34/524839952d5a58a7be9789e580bfc1ca883bf6579152444568
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24079 sha256=449e87ca835a6185aedb48201e24d61ed321b5e2a3ffb284196c83819333f3c2
      Stored in directory: /root/.cache/pip/wheels/34/31/15/7b070b25d0a549d20ce2e9fe6d727471c2c61ef904720fd40c
      Building wheel for docplex (setup.py) ... done
      Created wheel for docplex: filename=docplex-2.20.204-cp37-none-any.whl size=675362 sha256=bb9657b5981b38865d30d87ca199102f30d8e17b030048a825716b600f87c11c
      Stored in directory: /root/.cache/pip/wheels/ae/2c/e2/a099ebb6fda8adeba9c5fc2e25659d195ad2f5c6cc5fb75fd4
      Building wheel for dlx (setup.py) ... done
      Created wheel for dlx: filename=dlx-1.0.4-cp37-none-any.whl size=5712 sha256=2caaeb5548bc8c32b07ea3f4911709d212843a3229cace41e9cac51546df60ca
      Stored in directory: /root/.cache/pip/wheels/bb/ba/15/fdd0deb104df3254912998150ba9245668db06b00af5912d1a
      Building wheel for yfinance (setup.py) ... done
      Created wheel for yfinance: filename=yfinance-0.1.55-py2.py3-none-any.whl size=22616 sha256=cda9371b25d0220c339c4326516bbd505b991bfb53188273409113fbb84900ab
      Stored in directory: /root/.cache/pip/wheels/04/98/cc/2702a4242d60bdc14f48b4557c427ded1fe92aedf257d4565c
    Successfully built qiskit python-constraint docplex dlx yfinance
    ERROR: qiskit-aqua 0.9.1 has requirement retworkx<=0.8.0,>=0.7.0, but you'll have retworkx 0.9.0 which is incompatible.
    Installing collected packages: retworkx, ply, python-constraint, fastjsonschema, qiskit-terra, pybind11, qiskit-aer, websockets, ntlm-auth, cryptography, requests-ntlm, qiskit-ibmq-provider, qiskit-ignis, inflection, quandl, docplex, dlx, lxml, yfinance, qiskit-aqua, qiskit
      Found existing installation: lxml 4.2.6
        Uninstalling lxml-4.2.6:
          Successfully uninstalled lxml-4.2.6
    Successfully installed cryptography-3.4.7 dlx-1.0.4 docplex-2.20.204 fastjsonschema-2.15.1 inflection-0.5.1 lxml-4.6.3 ntlm-auth-1.5.0 ply-3.11 pybind11-2.6.2 python-constraint-1.4.0 qiskit-0.26.2 qiskit-aer-0.8.2 qiskit-aqua-0.9.1 qiskit-ibmq-provider-0.13.1 qiskit-ignis-0.6.0 qiskit-terra-0.17.4 quandl-3.6.0 requests-ntlm-1.1.0 retworkx-0.9.0 websockets-9.1 yfinance-0.1.55

```python
from qiskit.providers.aer import AerSimulator, QasmSimulator
from qiskit.algorithms import VQE
from qiskit.algorithms.optimizers import COBYLA
from qiskit.circuit.library import TwoLocal
# opflow is Qiskit's module for creating operators like yours
from qiskit import *
from qiskit.opflow import OperatorBase
from qiskit.opflow import Z, X, I  # Pauli Z, X matrices and identity
import pylab
import matplotlib.pyplot as plt
import numpy as np
```

```python
def store_intermediate_result(eval_count, parameters, mean, std):
  counts.append(eval_count)
  values.append(mean)

def get_gradient(values, counts):
    rvalues = []
    rcounts = []
    for i in range(1, len(values)):
        rvalues += [ (values[i - 1] - values[i]) ]
    rcounts = [ i for i in range(len(rvalues)) ]
    return rvalues, rcounts

def get_absolute(values):
    return [ (v if v > 0 else -v) for v in values]

```

```python
# J = 1.0
counts = []
values = []

H =   (504.0 * I^I^I^I^I^I^I^Z) + (1008.0 * I^I^I^I^I^I^Z^I) + (2016.0 * I^I^I^I^I^Z^I^I) + (504.0 * I^I^I^I^Z^I^I^I) + (1143.7999999999997 * I^I^I^Z^I^I^I^I) + (2287.6 * I^I^Z^I^I^I^I^I) + (4575.200000000001 * I^Z^I^I^I^I^I^I) + (1143.7999999999997 * Z^I^I^I^I^I^I^I) + (98.0 * I^I^I^I^I^I^Z^Z) + (196.0 * I^I^I^I^I^Z^I^Z) + (392.0 * I^I^I^I^I^Z^Z^I) + (49.0 * I^I^I^I^Z^I^I^Z) + (98.0 * I^I^I^I^Z^I^Z^I) + (196.0 * I^I^I^I^Z^Z^I^I) + (93.1 * I^I^Z^Z^I^I^I^I) + (186.2 * I^Z^I^Z^I^I^I^I) + (372.4 * I^Z^Z^I^I^I^I^I) + (46.55 * Z^I^I^Z^I^I^I^I) + (93.1 * Z^I^Z^I^I^I^I^I) + (186.2 * Z^Z^I^I^I^I^I^I)

# you can swap this for a real quantum device and keep the rest of the code the same!
# backend = AerSimulator(method='matrix_product_state')

backend = QasmSimulator()

# COBYLA usually works well for small problems like this one
optimizer = COBYLA(maxiter=2000)

# EfficientSU2 is a standard heuristic chemistry ansatz from Qiskit's circuit library
ansatz = TwoLocal(num_qubits=8, rotation_blocks=['ry', 'rz'], entanglement_blocks=None, entanglement='full', reps=1, skip_unentangled_qubits=False, skip_final_rotation_layer=True)
# set the algorithm
vqe = VQE(ansatz, optimizer, quantum_instance=backend, callback=store_intermediate_result)

#run it with the Hamiltonian we defined above
result = vqe.compute_minimum_eigenvalue(H)

#print the result (it contains lot's of information)
print(result)
```

    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 205,
        'eigenstate': {   '11011111': 0.03125,
                          '11110110': 0.04419417382415922,
                          '11110111': 0.1875,
                          '11111010': 0.03125,
                          '11111101': 0.03125,
                          '11111110': 0.125,
                          '11111111': 0.9717694878416383},
        'eigenvalue': -11153.551171875,
        'optimal_parameters': {   ParameterVectorElement(θ[1]): 3.046753482782516,
                                  ParameterVectorElement(θ[2]): 3.086210873485022,
                                  ParameterVectorElement(θ[0]): -2.90679844920079,
                                  ParameterVectorElement(θ[3]): -3.517745868482567,
                                  ParameterVectorElement(θ[5]): 3.0678710122000035,
                                  ParameterVectorElement(θ[8]): 7.900632289038266,
                                  ParameterVectorElement(θ[12]): -3.42596275263934,
                                  ParameterVectorElement(θ[10]): -0.2766063880344648,
                                  ParameterVectorElement(θ[6]): 3.128657974494508,
                                  ParameterVectorElement(θ[4]): -3.146939556266644,
                                  ParameterVectorElement(θ[9]): -1.596493351599472,
                                  ParameterVectorElement(θ[7]): 3.132115702422145,
                                  ParameterVectorElement(θ[11]): 2.1434175744649955,
                                  ParameterVectorElement(θ[14]): 1.4724479624926472,
                                  ParameterVectorElement(θ[13]): -2.124174149035791,
                                  ParameterVectorElement(θ[15]): -5.539345664240221},
        'optimal_point': array([-2.90679845, -0.27660639,  2.14341757, -3.42596275, -2.12417415,
            1.47244796, -5.53934566,  3.04675348,  3.08621087, -3.51774587,
           -3.14693956,  3.06787101,  3.12865797,  3.1321157 ,  7.90063229,
           -1.59649335]),
        'optimal_value': -11153.551171875,
        'optimizer_evals': 205,
        'optimizer_time': 5.959498167037964}

```python
v1, c1 = get_gradient(values, counts)
plt.figure(figsize=(15,15))
plt.plot(c1,v1)
plt.xlabel('optimization step')
plt.ylabel('E')
plt.show()
```

```python
counts = []
values = []

```

```python
H = (504.0 * I^I^I^I^I^I^I^Z) + (1008.0 * I^I^I^I^I^I^Z^I) +  (2016.0 * I^I^I^I^I^Z^I^I) + (504.0 * I^I^I^I^Z^I^I^I) +  (1143.7999999999997 * I^I^I^Z^I^I^I^I) +  (2287.6 * I^I^Z^I^I^I^I^I) +  (4575.200000000001 * I^Z^I^I^I^I^I^I) +  (1143.7999999999997 * Z^I^I^I^I^I^I^I) +  (98.0 * I^I^I^I^I^I^Z^Z) +  (196.0 * I^I^I^I^I^Z^I^Z) +  (392.0 * I^I^I^I^I^Z^Z^I) +  (49.0 * I^I^I^I^Z^I^I^Z) +  (98.0 * I^I^I^I^Z^I^Z^I) +  (196.0 * I^I^I^I^Z^Z^I^I) +  (93.1 * I^I^Z^Z^I^I^I^I) +  (186.2 * I^Z^I^Z^I^I^I^I) +  (372.4 * I^Z^Z^I^I^I^I^I) +  (46.55 * Z^I^I^Z^I^I^I^I) +  (93.1 * Z^I^Z^I^I^I^I^I) +  (186.2 * Z^Z^I^I^I^I^I^I)
# you can swap this for a real quantum device and keep the rest of the code the same!
# backend = AerSimulator(method='matrix_product_state')

backend = QasmSimulator()

# COBYLA usually works well for small problems like this one
optimizer = COBYLA(maxiter=2000)

# EfficientSU2 is a standard heuristic chemistry ansatz from Qiskit's circuit library
ansatz = TwoLocal(num_qubits=8, rotation_blocks=['ry', 'rz'], entanglement_blocks=None, entanglement='full', reps=1, skip_unentangled_qubits=False, skip_final_rotation_layer=True)
# set the algorithm
vqe = VQE(ansatz, optimizer, quantum_instance=backend, callback=store_intermediate_result)

#run it with the Hamiltonian we defined above
result = vqe.compute_minimum_eigenvalue(H)

#print the result (it contains lot's of information)
print(result)
```

    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 255,
        'eigenstate': {   '01101111': 0.03125,
                          '01111101': 0.03125,
                          '01111111': 0.05412658773652741,
                          '11101101': 0.03125,
                          '11101111': 0.32923917977664807,
                          '11111011': 0.03125,
                          '11111101': 0.10364452469860624,
                          '11111110': 0.03125,
                          '11111111': 0.9343697742328784},
        'eigenvalue': -10965.57451171875,
        'optimal_parameters': {   ParameterVectorElement(θ[5]): -3.165102845135186,
                                  ParameterVectorElement(θ[4]): 8.748449247035973,
                                  ParameterVectorElement(θ[3]): 3.1491153413860373,
                                  ParameterVectorElement(θ[2]): 3.2184621487031806,
                                  ParameterVectorElement(θ[1]): 2.9172726271812377,
                                  ParameterVectorElement(θ[0]): -3.233065992970459,
                                  ParameterVectorElement(θ[15]): -1.6486371526981256,
                                  ParameterVectorElement(θ[14]): -5.202419493700512,
                                  ParameterVectorElement(θ[13]): 2.66845475048185,
                                  ParameterVectorElement(θ[8]): 2.861947847378163,
                                  ParameterVectorElement(θ[7]): 3.0156205976029224,
                                  ParameterVectorElement(θ[9]): -5.806327276587659,
                                  ParameterVectorElement(θ[6]): 3.196220984325266,
                                  ParameterVectorElement(θ[12]): -4.927292036226532,
                                  ParameterVectorElement(θ[10]): -0.6884567108267369,
                                  ParameterVectorElement(θ[11]): -4.889955253296066},
        'optimal_point': array([-3.23306599, -0.68845671, -4.88995525, -4.92729204,  2.66845475,
           -5.20241949, -1.64863715,  2.91727263,  3.21846215,  3.14911534,
            8.74844925, -3.16510285,  3.19622098,  3.0156206 ,  2.86194785,
           -5.80632728]),
        'optimal_value': -10965.57451171875,
        'optimizer_evals': 255,
        'optimizer_time': 6.513612508773804}

```python
v2, c2 = get_gradient(values, counts)
plt.figure(figsize=(15,15))
plt.plot(c2,v2)
plt.xlabel('optimization step')
plt.ylabel('E')
plt.show()
```

```python
counts = []
values = []
```

```python
H = (504.0 * I^I^I^I^I^I^I^Z) + (1008.0 * I^I^I^I^I^I^Z^I) + (2016.0 * I^I^I^I^I^Z^I^I) + (504.0 * I^I^I^I^Z^I^I^I) + (1143.7999999999997 * I^I^I^Z^I^I^I^I) + (2287.6 * I^I^Z^I^I^I^I^I) + (4575.200000000001 * I^Z^I^I^I^I^I^I) + (1143.7999999999997 * Z^I^I^I^I^I^I^I) + (98.0 * I^I^I^I^I^I^Z^Z)+(196.0 * I^I^I^I^I^Z^I^Z) + (392.0 * I^I^I^I^I^Z^Z^I) +  (49.0 * I^I^I^I^Z^I^I^Z) +  (98.0 * I^I^I^I^Z^I^Z^I) +  (196.0 * I^I^I^I^Z^Z^I^I) +  (93.1 * I^I^Z^Z^I^I^I^I) +  (186.2 * I^Z^I^Z^I^I^I^I)+  (372.4 * I^Z^Z^I^I^I^I^I) +  (46.55 * Z^I^I^Z^I^I^I^I) +  (93.1 * Z^I^Z^I^I^I^I^I) +  (186.2 * Z^Z^I^I^I^I^I^I)

# you can swap this for a real quantum device and keep the rest of the code the same!
# backend = AerSimulator(method='matrix_product_state')

backend = QasmSimulator()

# COBYLA usually works well for small problems like this one
optimizer = COBYLA(maxiter=2000)

# EfficientSU2 is a standard heuristic chemistry ansatz from Qiskit's circuit library
ansatz = TwoLocal(num_qubits=8, rotation_blocks=['ry', 'rz'], entanglement_blocks=None, entanglement='full', reps=1, skip_unentangled_qubits=False, skip_final_rotation_layer=True)
# set the algorithm
vqe = VQE(ansatz, optimizer, quantum_instance=backend, callback=store_intermediate_result)

#run it with the Hamiltonian we defined above
result = vqe.compute_minimum_eigenvalue(H)

#print the result (it contains lot's of information)
print(result)
```

    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 214,
        'eigenstate': {   '01111110': 0.04419417382415922,
                          '01111111': 0.08267972847076846,
                          '10111110': 0.04419417382415922,
                          '10111111': 0.03125,
                          '11101111': 0.03125,
                          '11110110': 0.2049199538844375,
                          '11110111': 0.3321920566479578,
                          '11111011': 0.03125,
                          '11111110': 0.46351240544347894,
                          '11111111': 0.7868548945008857},
        'eigenvalue': -10992.766503906249,
        'optimal_parameters': {   ParameterVectorElement(θ[2]): 3.0754720787264387,
                                  ParameterVectorElement(θ[0]): -2.0394715204534624,
                                  ParameterVectorElement(θ[1]): 3.1508103200825435,
                                  ParameterVectorElement(θ[10]): -1.6945307029822305,
                                  ParameterVectorElement(θ[14]): -0.49360231223732737,
                                  ParameterVectorElement(θ[13]): -4.6801055701797445,
                                  ParameterVectorElement(θ[9]): 4.153760587461629,
                                  ParameterVectorElement(θ[11]): 6.018932414251311,
                                  ParameterVectorElement(θ[12]): 3.2037597470228123,
                                  ParameterVectorElement(θ[8]): 1.5798555821165452,
                                  ParameterVectorElement(θ[6]): -3.2219695042036944,
                                  ParameterVectorElement(θ[7]): -2.9484899978240806,
                                  ParameterVectorElement(θ[5]): 3.1941277844827902,
                                  ParameterVectorElement(θ[4]): -3.1597007294710098,
                                  ParameterVectorElement(θ[3]): -3.891670254356035,
                                  ParameterVectorElement(θ[15]): -1.2843062805907928},
        'optimal_point': array([-2.03947152, -1.6945307 ,  6.01893241,  3.20375975, -4.68010557,
           -0.49360231, -1.28430628,  3.15081032,  3.07547208, -3.89167025,
           -3.15970073,  3.19412778, -3.2219695 , -2.94849   ,  1.57985558,
            4.15376059]),
        'optimal_value': -10992.766503906249,
        'optimizer_evals': 214,
        'optimizer_time': 7.787711143493652}

```python
v3, c3 = get_gradient(values, counts)
plt.figure(figsize=(15,15))
plt.plot(c3,v3)
plt.xlabel('optimization step')
plt.ylabel('E')
plt.show()
```

```python
counts = []
values = []
```

```python
H = (504.0 * I^I^I^I^I^I^I^Z) +  (1008.0 * I^I^I^I^I^I^Z^I) +  (2016.0 * I^I^I^I^I^Z^I^I) +  (504.0 * I^I^I^I^Z^I^I^I) +  (1143.7999999999997 * I^I^I^Z^I^I^I^I) +  (2287.6 * I^I^Z^I^I^I^I^I) +  (4575.200000000001 * I^Z^I^I^I^I^I^I) +  (1143.7999999999997 * Z^I^I^I^I^I^I^I)+  (98.0 * I^I^I^I^I^I^Z^Z) +  (196.0 * I^I^I^I^I^Z^I^Z) +  (392.0 * I^I^I^I^I^Z^Z^I) +  (49.0 * I^I^I^I^Z^I^I^Z)+  (98.0 * I^I^I^I^Z^I^Z^I)+  (196.0 * I^I^I^I^Z^Z^I^I) +  (93.1 * I^I^Z^Z^I^I^I^I)+  (186.2 * I^Z^I^Z^I^I^I^I)+  (372.4 * I^Z^Z^I^I^I^I^I)+  (46.55 * Z^I^I^Z^I^I^I^I)+  (93.1 * Z^I^Z^I^I^I^I^I)+  (186.2 * Z^Z^I^I^I^I^I^I)

# you can swap this for a real quantum device and keep the rest of the code the same!
# backend = AerSimulator(method='matrix_product_state')

backend = QasmSimulator()

# COBYLA usually works well for small problems like this one
optimizer = COBYLA(maxiter=2000)

# EfficientSU2 is a standard heuristic chemistry ansatz from Qiskit's circuit library
ansatz = TwoLocal(num_qubits=8, rotation_blocks=['ry', 'rz'], entanglement_blocks=None, entanglement='full', reps=1, skip_unentangled_qubits=False, skip_final_rotation_layer=True)
# set the algorithm
vqe = VQE(ansatz, optimizer, quantum_instance=backend, callback=store_intermediate_result)

#run it with the Hamiltonian we defined above
result = vqe.compute_minimum_eigenvalue(H)

#print the result (it contains lot's of information)
print(result)
```

    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 184,
        'eigenstate': {   '11100111': 0.03125,
                          '11101110': 0.03125,
                          '11110011': 0.04419417382415922,
                          '11110110': 0.23385358667337133,
                          '11110111': 0.649519052838329,
                          '11111010': 0.03125,
                          '11111011': 0.03125,
                          '11111101': 0.05412658773652741,
                          '11111110': 0.24206145913796356,
                          '11111111': 0.6753182120304472},
        'eigenvalue': -10963.80126953125,
        'optimal_parameters': {   ParameterVectorElement(θ[7]): -3.0764720942752097,
                                  ParameterVectorElement(θ[9]): -2.9562550418913713,
                                  ParameterVectorElement(θ[6]): 3.1672992325151417,
                                  ParameterVectorElement(θ[4]): -3.0813931010365248,
                                  ParameterVectorElement(θ[8]): 2.104987703370218,
                                  ParameterVectorElement(θ[12]): -5.736766438853289,
                                  ParameterVectorElement(θ[2]): -3.047317376298203,
                                  ParameterVectorElement(θ[10]): 0.6228108551453106,
                                  ParameterVectorElement(θ[11]): 7.221157300557545,
                                  ParameterVectorElement(θ[5]): 3.1023633907103267,
                                  ParameterVectorElement(θ[3]): 4.643156654705857,
                                  ParameterVectorElement(θ[14]): -3.4173058051282617,
                                  ParameterVectorElement(θ[13]): 5.145241204885928,
                                  ParameterVectorElement(θ[15]): -3.0333728595768443,
                                  ParameterVectorElement(θ[1]): 3.0637009227102765,
                                  ParameterVectorElement(θ[0]): 3.849449691053329},
        'optimal_point': array([ 3.84944969,  0.62281086,  7.2211573 , -5.73676644,  5.1452412 ,
           -3.41730581, -3.03337286,  3.06370092, -3.04731738,  4.64315665,
           -3.0813931 ,  3.10236339,  3.16729923, -3.07647209,  2.1049877 ,
           -2.95625504]),
        'optimal_value': -10963.80126953125,
        'optimizer_evals': 184,
        'optimizer_time': 6.882545471191406}

```python
v4, c4 = get_gradient(values, counts)
plt.figure(figsize=(15,15))
plt.plot(c4,v4)
plt.xlabel('optimization step')
plt.ylabel('E')
plt.show()
```

```python
# 2nd Order Der

v1, c1 = get_gradient(v1, c1)
v2, c2 = get_gradient(v2, c2)
v3, c3 = get_gradient(v3, c3)
v4, c4 = get_gradient(v4, c4)

```

```python
v1 = get_absolute(v1)
v2 = get_absolute(v2)
v3 = get_absolute(v3)
v4 = get_absolute(v4)
```

```python
plt.figure(figsize=(15,15))
plt.plot(c1,v1, label="R1")
plt.plot(c2,v2, label="R2")
plt.plot(c3,v3, label="R3")
plt.plot(c4,v4, label="R4")
plt.xlabel('comparison optimization step')
plt.ylabel('E')
plt.legend()
plt.show()
```

```python
plt.figure(figsize=(15,15))
plt.plot(c1[:100],v1[:100], label="R1")
plt.plot(c2[:100],v2[:100], label="R2")
plt.plot(c3[:100],v3[:100], label="R3")
plt.plot(c4[:100],v4[:100], label="R4")
plt.xlabel('comparison optimization step')
plt.ylabel('E')
plt.legend()
plt.show()
```

```python

```


- title: Active Perception in Adversarial Scenarios
- summary: Accumulating evidence about peers to discriminate potential threats.
- author: Daniel Cox
- date: 2019-09-22
- category: arXiv highlights
- image: /static/images/arXiv.gif

# This week

This week's paper is [Active Perception in Adversarial Scenarios using Maximum Entropy Deep Reinforcement Learning](https://arxiv.org/abs/1902.05644v1). The idea is that an agent interacting with another agent can learn to assess the threat it may pose. It does this by actively testing the opponent agent's behavior, and does not assume the opponent's behavior remains stationary. It uses Bayesian filtering to update its belief about the disposition of the opponent, and that's why this paper caught my eye. I'm on a Bayesian kick lately.

> To summarize, the contribution here is the development of a scalable robust active perception method in scenarios where a potential adversary opponent could be actively hostile to the intent recognition activity, which extends and outperforms the POMDP methods.

I'm a bit short on time this week, so I apologize for the amount of jargon and the unusually high level of confusion.

# Problem setup

> We model the active perception problem as a planning problem, defined by the tuple $\langle S,A^a,A^o,T,O,R,b_0,\gamma \rangle$, where $S=\langle S^o,S^p \rangle$ is the state of the world, consisting of the set of observable states $S^o$ and the set of partially observable states $S^p$; $A^a$ is the set of actions of the autonomous agent; $A^o$ is the set of actions of the opponent; we further assume that regardless of the intention, the opponent has the same set of observable actions. Otherwise, an intention is easily identifiable once an action that is uniquely corresponding to that type of intention is observed. $T:S \times A^a \times A^o \rightarrow \Delta_S $ is the transition probability, where $\Delta_{\bullet}$ denotes the space of probability distribution over the space $\bullet$. $O: S \times A^a \rightarrow \Delta_{A^o}$ is the observation probability; $R: S \times A^a \times A^o \rightarrow  \mathbb{R}$ is the reward function; $b_0$ is the prior probability of the opponent being an adversary; and $\gamma$ is the discount factor.

Further, the opponent is assumed to be either neutral (merely self-interested, in a known way) or hostile (goal-directed, as defined by a known MDP), with bounded rationality, (it may not be able to take the optimal action) and it is likely to behave deceptively.

Notice that the actual behavior of the opponent is known if its disposition is known, which to my mind may or may not be a reasonable assumption, depending on the setting. Since I've had AI safety on the brain lately, it strikes me as _unrealistic_ in a situation where your opponent is smarter than you are. It may be more realistic in settings where everyone has the same goal and it's relatively clear how anyway would try to achieve it if they didn't have to deal with other agents.

The authors' adversarial model is interesting. ($\lambda$ is the parameter to $\pi^o$ that specifies whether the agent is neutral: $\lambda=0$, or adversarial: $\lambda=1$):

> We use the following equation to model an adversarial agent's policy $\pi^o$:
$$
\begin{align}
\pi^o(a^o_t|s_t,\lambda=1;\alpha,\beta)= & \text{argmin}_{\pi \in \Delta} \{\mathbb{KL}(\pi|\pi^{\text{MDP}}_{\alpha})\\
& +\beta \mathbb{KL}(\pi|\pi^o(\cdot|s_t,\lambda=0)) \}
\pi^{\text{MDP}}_{\alpha}(a_t^o|s_t,\lambda=1)=e^{\alpha Q(s_t,a_t^o)}/Z(s_t)
\end{align}
$$

The thing to take away from this is that both rationality and deception are tunable parameters. The rationality of the opponent is controlled by the temperature parameter $\alpha$, by adjusting how well the opponent makes use of the optimal Q function. The degree to which the opponent is _deceptive_ is controlled by $\beta$, which adjusts how much the KL-divergence of the existing policy from the _neutral_ policy affects the opponent's search for an optimal strategy.

# Bayesian filtering

> We maintain a belief $b_t(\lambda)$ over the hidden variable by Bayesian filtering.

As I mentioned, I'm rather short on time today, so I must apologize again for not actually spending the time to explain this. For now, suffice it to say that the opponent is either neutral ($\lambda=0$) or hostile ($\lambda=1$), and how your agent reacts to it depends very much on which one of those it believes it is playing against. Bayesian filtering will allow it to make the most of the evidence available, so it can use its best guess as it trains.

> We define a hybrid belief-state dependent reward to balance exploration and safety
\begin{equation}
\begin{aligned}
r(b_t,s_t,a^a_t)&=-H(b_t)+r(s_t,a^a_t)\\
&=b\log b+(1-b)\log(1-b)+r(s_t,a^a_t),
\end{aligned}
\label{eq6}
\end{equation}
where we use the shorthand $b$ to denote $b_t(\lambda=1)$, the belief that the opponent is an adversary; and $r(s_t,a^a_t)$ is the state dependent reward.

>This reward balances exploration behavior and safety. The negative entropy reward $-H(b_t)$ can be interpreted as maximizing the expected logarithm of true positive rate (TPR) and true negative rate (TNR). The state-dependent reward $r(s_t,a^a_t)$ depends both on the observable state and the partially observable intent state $\lambda$, as well as the action of the autonomous agent. This reward
is used to ensure safety. For instance, some actions could be dangerous to the neutral [opponent], which are discouraged by a large negative reward.

Our agent is trained using [Soft-Q Learning](https://arxiv.org/abs/1702.08165) while values of $\lambda$ are varied, with corresponding opponent behavior. Interestingly, in the case study section the authors mention that the actual adversary models were not always provided in the learning phase.

> The active perception agent has to identify the hidden intent while bein grobust to this model uncertainty, which is challenging.

# Parting thoughts

1. I admit to being a bit confused by this paper. The authors claim to do Bayesian filtering, but it's not an explicit feature of the algorithm. In fact, they seem to be sampling $\lambda$ for use in training by using only $b_0$, their prior probability for their belief state. Perhaps it's a typo.
2. They also seem to claim that the two models of the opponent behavior must be known, but then they mention they're not available during the learning phase in their case study. Drop me a line if this makes sense to you.

content/Active Perception in Adversarial Scenarios.ipynb