### Install Qiskit

```
pip install qiskit
```

    Collecting qiskit
      Downloading https://files.pythonhosted.org/packages/69/3d/e8f748f6e1b54714c691f6bb95c986c236576386d7a86c6b737dd84b281b/qiskit-0.11.0.tar.gz
    Collecting qiskit_terra==0.8.2 (from qiskit)
      Downloading https://files.pythonhosted.org/packages/71/52/ebfa1c41722ae5e04d2456677e6d492373d9b28eca5cdf86b8f2981b1e21/qiskit_terra-0.8.2-cp36-cp36m-manylinux1_x86_64.whl (1.5MB)
         |████████████████████████████████| 1.5MB 5.1MB/s
    Collecting qiskit-aer==0.2.3 (from qiskit)
      Downloading https://files.pythonhosted.org/packages/1c/f6/10bb8a7b8939623a5ffedc844f7bba535b00a014292a58a82893e966bbde/qiskit_aer-0.2.3-cp36-cp36m-manylinux1_x86_64.whl (21.3MB)
         |████████████████████████████████| 21.3MB 1.0MB/s
    Collecting qiskit-ibmq-provider==0.3.0 (from qiskit)
      Downloading https://files.pythonhosted.org/packages/90/b9/fd1997a20bc26e689b2987e196779ae8ce8c8052fc2a49baa50edfda8d99/qiskit-ibmq-provider-0.3.0.tar.gz (56kB)
         |████████████████████████████████| 61kB 22.3MB/s
    Collecting qiskit-ignis==0.1.1 (from qiskit)
      Downloading https://files.pythonhosted.org/packages/6e/4e/77230b3a51a864fd54c2ef7b6a0d417063cf4b18943ce8fcd45e15f74f3e/qiskit_ignis-0.1.1-py3-none-any.whl (78kB)
         |████████████████████████████████| 81kB 27.2MB/s
    Collecting qiskit-aqua==0.5.2 (from qiskit)
      Downloading https://files.pythonhosted.org/packages/3d/2c/18f987e2b673b0c47f6293f6d2c51e1f6db176744c81321f6cf28614d458/qiskit_aqua-0.5.2-py3-none-any.whl (401kB)
         |████████████████████████████████| 409kB 44.9MB/s
    Collecting qiskit-chemistry==0.5.0 (from qiskit)
      Downloading https://files.pythonhosted.org/packages/41/35/6b10aa3ea2fbe1b64cb0c3b2eaecb23cff044b574b151e8adc9ab9fbfe9a/qiskit_chemistry-0.5.0-py3-none-any.whl (516kB)
         |████████████████████████████████| 522kB 40.5MB/s
    Requirement already satisfied: jsonschema<2.7,>=2.6 in /usr/local/lib/python3.6/dist-packages (from qiskit_terra==0.8.2->qiskit) (2.6.0)
    Collecting pylatexenc>=1.4 (from qiskit_terra==0.8.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/2d/44/93573fef5cb46842ce9a87548008bea92c8f7523c7c2152cfdffee269ed5/pylatexenc-1.5.tar.gz (47kB)
         |████████████████████████████████| 51kB 19.1MB/s
    Collecting sympy>=1.3 (from qiskit_terra==0.8.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/21/21/f4105795ca7f35c541d82c5b06be684dd2f5cb4f508fb487cd7aea4de776/sympy-1.4-py2.py3-none-any.whl (5.3MB)
         |████████████████████████████████| 5.3MB 32.7MB/s
    Collecting marshmallow<3,>=2.17.0 (from qiskit_terra==0.8.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/77/0b/ee5a4d0b7bc88bb3b59a25fb40e1811810c039f7faa1e87dcdcbcb1fdc33/marshmallow-2.19.5-py2.py3-none-any.whl (50kB)
         |████████████████████████████████| 51kB 19.1MB/s
    Collecting marshmallow-polyfield<4,>=3.2 (from qiskit_terra==0.8.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/fa/ba/6406b8cce3e42b2838d234310c3605627083bbb8f71dcc312c616480fe92/marshmallow_polyfield-3.2-py2.py3-none-any.whl
    Collecting ply>=3.10 (from qiskit_terra==0.8.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
         |████████████████████████████████| 51kB 21.2MB/s
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.6/dist-packages (from qiskit_terra==0.8.2->qiskit) (1.3.0)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.6/dist-packages (from qiskit_terra==0.8.2->qiskit) (5.4.8)
    Requirement already satisfied: numpy>=1.13 in /usr/local/lib/python3.6/dist-packages (from qiskit_terra==0.8.2->qiskit) (1.16.4)
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.6/dist-packages (from qiskit_terra==0.8.2->qiskit) (2.3)
    Requirement already satisfied: pillow>=4.2.1 in /usr/local/lib/python3.6/dist-packages (from qiskit_terra==0.8.2->qiskit) (4.3.0)
    Collecting nest-asyncio>=1.0.0 (from qiskit-ibmq-provider==0.3.0->qiskit)
      Downloading https://files.pythonhosted.org/packages/62/18/4c6a895c925e8822e72535d774861e9ca07089c4b938ed20002b1aaab66a/nest_asyncio-1.0.0-py3-none-any.whl
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.6/dist-packages (from qiskit-ibmq-provider==0.3.0->qiskit) (2.21.0)
    Collecting requests-ntlm>=1.1.0 (from qiskit-ibmq-provider==0.3.0->qiskit)
      Downloading https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
    Collecting websockets<8,>=7 (from qiskit-ibmq-provider==0.3.0->qiskit)
      Downloading https://files.pythonhosted.org/packages/43/71/8bfa882b9c502c36e5c9ef6732969533670d2b039cbf95a82ced8f762b80/websockets-7.0-cp36-cp36m-manylinux1_x86_64.whl (63kB)
         |████████████████████████████████| 71kB 24.3MB/s
    Collecting docplex (from qiskit-aqua==0.5.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/06/bc/b17191a0f8595763aba75c4ae5cc0f44be594c8d75aed737c42cf167c73f/docplex-2.10.150.tar.gz (528kB)
         |████████████████████████████████| 532kB 46.4MB/s
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.5.2->qiskit) (41.0.1)
    Collecting dlx (from qiskit-aqua==0.5.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/54/c0/b8fb5bb727e983b6f5251433ef941b48f38c65bb0bd6ec509e9185bcd406/dlx-1.0.4.tar.gz
    Collecting quandl (from qiskit-aqua==0.5.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/12/64/6314547cb01fa75697ddba6d6d43a9f1d023c1730ffc6abf3db7a0c89222/Quandl-3.4.8-py2.py3-none-any.whl
    Requirement already satisfied: fastdtw in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.5.2->qiskit) (0.3.2)
    Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.5.2->qiskit) (0.21.2)
    Requirement already satisfied: cvxopt in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.5.2->qiskit) (1.2.3)
    Collecting pyscf; sys_platform != "win32" (from qiskit-chemistry==0.5.0->qiskit)
      Downloading https://files.pythonhosted.org/packages/db/4b/2a82e6f6dc24fceaf2cb1e2537796b0e9b7f3187d8008ef1f48e2cacb728/pyscf-1.6.2-cp36-cp36m-manylinux1_x86_64.whl (21.5MB)
         |████████████████████████████████| 21.5MB 22.6MB/s
    Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from qiskit-chemistry==0.5.0->qiskit) (2.8.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy>=1.3->qiskit_terra==0.8.2->qiskit) (1.1.0)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.2->qiskit_terra==0.8.2->qiskit) (4.4.0)
    Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow>=4.2.1->qiskit_terra==0.8.2->qiskit) (0.46)
    Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.3.0->qiskit) (1.24.3)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.3.0->qiskit) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.3.0->qiskit) (2019.6.16)
    Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.3.0->qiskit) (2.8)
    Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.0->qiskit)
      Downloading https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
    Collecting cryptography>=1.3 (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.0->qiskit)
      Downloading https://files.pythonhosted.org/packages/97/18/c6557f63a6abde34707196fb2cad1c6dc0dbff25a200d5044922496668a4/cryptography-2.7-cp34-abi3-manylinux1_x86_64.whl (2.3MB)
         |████████████████████████████████| 2.3MB 32.2MB/s
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from docplex->qiskit-aqua==0.5.2->qiskit) (1.12.0)
    Collecting docloud>=1.0.369 (from docplex->qiskit-aqua==0.5.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/89/1c/fcb7ea86ae5cb7137a22f00bcaa785de2d102bd4a2b16383ec4a877a835b/docloud-1.0.375.tar.gz
    Collecting more-itertools<=5.0.0 (from quandl->qiskit-aqua==0.5.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/a4/a6/42f17d065bda1fac255db13afc94c93dbfb64393eae37c749b4cb0752fc7/more_itertools-5.0.0-py3-none-any.whl (52kB)
         |████████████████████████████████| 61kB 22.9MB/s
    Collecting ndg-httpsclient (from quandl->qiskit-aqua==0.5.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/fb/67/c2f508c00ed2a6911541494504b7cac16fe0b0473912568df65fd1801132/ndg_httpsclient-0.5.1-py3-none-any.whl
    Requirement already satisfied: pandas>=0.14 in /usr/local/lib/python3.6/dist-packages (from quandl->qiskit-aqua==0.5.2->qiskit) (0.24.2)
    Requirement already satisfied: pyasn1 in /usr/local/lib/python3.6/dist-packages (from quandl->qiskit-aqua==0.5.2->qiskit) (0.4.5)
    Collecting pyOpenSSL (from quandl->qiskit-aqua==0.5.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)
         |████████████████████████████████| 61kB 23.3MB/s
    Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/dist-packages (from quandl->qiskit-aqua==0.5.2->qiskit) (2.5.3)
    Collecting inflection>=0.3.1 (from quandl->qiskit-aqua==0.5.2->qiskit)
      Downloading https://files.pythonhosted.org/packages/d5/35/a6eb45b4e2356fe688b21570864d4aa0d0a880ce387defe9c589112077f8/inflection-0.3.1.tar.gz
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.5.2->qiskit) (0.13.2)
    Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.0->qiskit)
      Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
         |████████████████████████████████| 102kB 29.0MB/s
    Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.6/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.0->qiskit) (1.12.3)
    Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.14->quandl->qiskit-aqua==0.5.2->qiskit) (2018.9)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.0->qiskit) (2.19)
    Building wheels for collected packages: qiskit, qiskit-ibmq-provider, pylatexenc, docplex, dlx, docloud, inflection
      Building wheel for qiskit (setup.py) ... error
      ERROR: Failed building wheel for qiskit
      Running setup.py clean for qiskit
      Building wheel for qiskit-ibmq-provider (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/0c/56/e7/b07b4d2f19fd4c5fff7fe334c72eb313fecf04fa693a7c558e
      Building wheel for pylatexenc (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/da/d1/f1/fa2d005ecda8836039af952de8f4bf1defd42fe385027b6f52
      Building wheel for docplex (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/6b/9d/3a/25dc38df1a476f24a36f1f3c285b696a28273d00124b571c08
      Building wheel for dlx (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/bb/ba/15/fdd0deb104df3254912998150ba9245668db06b00af5912d1a
      Building wheel for docloud (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/4e/24/19/9b0e57545d7543ecec35c5faff78c058bf0c2506369c8b1d31
      Building wheel for inflection (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/9f/5a/d3/6fc3bf6516d2a3eb7e18f9f28b472110b59325f3f258fe9211
    Successfully built qiskit-ibmq-provider pylatexenc docplex dlx docloud inflection
    Failed to build qiskit
    ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.
    Installing collected packages: pylatexenc, sympy, marshmallow, marshmallow-polyfield, ply, qiskit-terra, qiskit-aer, nest-asyncio, ntlm-auth, asn1crypto, cryptography, requests-ntlm, websockets, qiskit-ibmq-provider, qiskit-ignis, docloud, docplex, dlx, more-itertools, pyOpenSSL, ndg-httpsclient, inflection, quandl, qiskit-aqua, pyscf, qiskit-chemistry, qiskit
      Found existing installation: sympy 1.1.1
        Uninstalling sympy-1.1.1:
          Successfully uninstalled sympy-1.1.1
      Found existing installation: more-itertools 7.1.0
        Uninstalling more-itertools-7.1.0:
          Successfully uninstalled more-itertools-7.1.0
      Running setup.py install for qiskit ... done
    Successfully installed asn1crypto-0.24.0 cryptography-2.7 dlx-1.0.4 docloud-1.0.375 docplex-2.10.150 inflection-0.3.1 marshmallow-2.19.5 marshmallow-polyfield-3.2 more-itertools-5.0.0 ndg-httpsclient-0.5.1 nest-asyncio-1.0.0 ntlm-auth-1.3.0 ply-3.11 pyOpenSSL-19.0.0 pylatexenc-1.5 pyscf-1.6.2 qiskit-0.11.0 qiskit-aer-0.2.3 qiskit-aqua-0.5.2 qiskit-chemistry-0.5.0 qiskit-ibmq-provider-0.3.0 qiskit-ignis-0.1.1 qiskit-terra-0.8.2 quandl-3.4.8 requests-ntlm-1.1.0 sympy-1.4 websockets-7.0

```
# import common packages
import numpy as np

from qiskit import Aer

# lib from Qiskit Aqua
from qiskit.aqua import Operator, QuantumInstance
from qiskit.aqua.algorithms import VQE, ExactEigensolver
from qiskit.aqua.components.optimizers import COBYLA

# lib from Qiskit Aqua Chemistry
from qiskit.chemistry import FermionicOperator
from qiskit.chemistry.drivers import PySCFDriver, UnitsType
from qiskit.chemistry.aqua_extensions.components.variational_forms import UCCSD
from qiskit.chemistry.aqua_extensions.components.initial_states import HartreeFock
```

### Set up the molecular structure

```
driver = PySCFDriver(atom='H .0 .0 .0; H .0 .0 0.72', unit=UnitsType.ANGSTROM, basis='sto3g')
molecule = driver.run()
```

```
h1 = molecule.one_body_integrals
h2 = molecule.two_body_integrals
nuclear_repulsion_energy = molecule.nuclear_repulsion_energy

num_particles = molecule.num_alpha + molecule.num_beta
num_spin_orbitals = molecule.num_orbitals * 2
print("HF energy: {}".format(molecule.hf_energy))
print("# of electrons: {}".format(num_particles))
print("# of spin orbitals: {}".format(num_spin_orbitals))
```

    HF energy: -1.1174450773714701
    # of electrons: 2
    # of spin orbitals: 4

### Choose your mapping

```
map_type = 'jordan_wigner'
```

### Generate the Pauli strings

```
ferOp = FermionicOperator(h1=h1, h2=h2)
qubitOp = ferOp.mapping(map_type=map_type, threshold=0.00000001)

print(qubitOp.print_operators())
print(qubitOp)
```

    IIII	(-0.8362123048044218+0j)
    IIIZ	(0.08705553358137133+0j)
    IIZI	(-0.024253213726949913+0j)
    IZII	(0.08705553358137129+0j)
    ZIII	(-0.02425321372694993+0j)
    IIZZ	(0.07854309838088275+0j)
    IZIZ	(0.13546887571547914+0j)
    XXYY	(0.05897532134962703+0j)
    YYYY	(0.05897532134962703+0j)
    XXXX	(0.05897532134962703+0j)
    YYXX	(0.05897532134962703+0j)
    ZIIZ	(0.1375184197305098+0j)
    IZZI	(0.1375184197305098+0j)
    ZIZI	(0.14301575315731474+0j)
    ZZII	(0.07854309838088275+0j)

    Representation: paulis, qubits: 4, size: 15

### Set up the Hartree Fock states

```
qubit_reduction = False
HF_state = HartreeFock(qubitOp.num_qubits, num_spin_orbitals, num_particles, map_type,
                       qubit_reduction)
```

### Set up the UCCSD ansatzs

```
uccsd_ansatz = UCCSD(qubitOp.num_qubits, depth=1,
                   num_orbitals=num_spin_orbitals, num_particles=num_particles,
                   initial_state=HF_state, qubit_mapping=map_type,
                   two_qubit_reduction=qubit_reduction)
list = np.random.rand(3)
print(list)
uccsd_ansatz_circuit = uccsd_ansatz.construct_circuit(list)
```

    [0.8492349  0.28429897 0.57207906]

### Visualize the UCCSD ansatz circuit

```
print(uccsd_ansatz_circuit)
```

            ┌─────────────────────┐┌──────────┐                         »
    q_0: |0>┤ U3(3.1416,0,3.1416) ├┤ U2(0,pi) ├──■───────────────────■──»
            ├─────────────────────┤└──────────┘┌─┴─┐┌─────────────┐┌─┴─┐»
    q_1: |0>┤ U3(pi/2,-pi/2,pi/2) ├────────────┤ X ├┤ U1(0.84923) ├┤ X ├»
            ├─────────────────────┤┌──────────┐└───┘└─────────────┘└───┘»
    q_2: |0>┤ U3(3.1416,0,3.1416) ├┤ U2(0,pi) ├──■───────────────────■──»
            ├─────────────────────┤└──────────┘┌─┴─┐ ┌────────────┐┌─┴─┐»
    q_3: |0>┤ U3(pi/2,-pi/2,pi/2) ├────────────┤ X ├─┤ U1(0.2843) ├┤ X ├»
            └─────────────────────┘            └───┘ └────────────┘└───┘»
    «           ┌──────────┐      ┌─────────────────────┐                          »
    «q_0: ──────┤ U2(0,pi) ├──────┤ U3(pi/2,-pi/2,pi/2) ├──■────────────────────■──»
    «     ┌─────┴──────────┴─────┐└─────┬──────────┬────┘┌─┴─┐┌──────────────┐┌─┴─┐»
    «q_1: ┤ U3(-pi/2,-pi/2,pi/2) ├──────┤ U2(0,pi) ├─────┤ X ├┤ U1(-0.84923) ├┤ X ├»
    «     └─────┬──────────┬─────┘┌─────┴──────────┴────┐└───┘└──────────────┘└───┘»
    «q_2: ──────┤ U2(0,pi) ├──────┤ U3(pi/2,-pi/2,pi/2) ├──■────────────────────■──»
    «     ┌─────┴──────────┴─────┐└─────┬──────────┬────┘┌─┴─┐┌─────────────┐ ┌─┴─┐»
    «q_3: ┤ U3(-pi/2,-pi/2,pi/2) ├──────┤ U2(0,pi) ├─────┤ X ├┤ U1(-0.2843) ├─┤ X ├»
    «     └──────────────────────┘      └──────────┘     └───┘└─────────────┘ └───┘»
    «     ┌──────────────────────┐      ┌──────────┐                    »
    «q_0: ┤ U3(-pi/2,-pi/2,pi/2) ├──────┤ U2(0,pi) ├───────■────────────»
    «     └─────┬──────────┬─────┘      ├──────────┤     ┌─┴─┐          »
    «q_1: ──────┤ U2(0,pi) ├────────────┤ U2(0,pi) ├─────┤ X ├──■───────»
    «     ┌─────┴──────────┴─────┐      ├──────────┤     └───┘┌─┴─┐     »
    «q_2: ┤ U3(-pi/2,-pi/2,pi/2) ├──────┤ U2(0,pi) ├──────────┤ X ├──■──»
    «     └─────┬──────────┬─────┘┌─────┴──────────┴────┐     └───┘┌─┴─┐»
    «q_3: ──────┤ U2(0,pi) ├──────┤ U3(pi/2,-pi/2,pi/2) ├──────────┤ X ├»
    «           └──────────┘      └─────────────────────┘          └───┘»
    «                                                             ┌──────────┐»
    «q_0: ─────────────────────────────────────────────────■──────┤ U2(0,pi) ├»
    «                                                    ┌─┴─┐    ├──────────┤»
    «q_1: ───────────────────────────────■───────────────┤ X ├────┤ U2(0,pi) ├»
    «                                  ┌─┴─┐          ┌──┴───┴───┐├──────────┤»
    «q_2: ─────────────────■───────────┤ X ├──────────┤ U2(0,pi) ├┤ U2(0,pi) ├»
    «     ┌─────────────┐┌─┴─┐┌────────┴───┴─────────┐├──────────┤└──────────┘»
    «q_3: ┤ U1(0.14302) ├┤ X ├┤ U3(-pi/2,-pi/2,pi/2) ├┤ U2(0,pi) ├────────────»
    «     └─────────────┘└───┘└──────────────────────┘└──────────┘            »
    «           ┌──────────┐                                                    »
    «q_0: ──────┤ U2(0,pi) ├───────■────────────────────────────────────────────»
    «     ┌─────┴──────────┴────┐┌─┴─┐                                          »
    «q_1: ┤ U3(pi/2,-pi/2,pi/2) ├┤ X ├──■────────────────────────────────■──────»
    «     └─────────────────────┘└───┘┌─┴─┐                            ┌─┴─┐    »
    «q_2: ────────────────────────────┤ X ├──■───────────────────■─────┤ X ├────»
    «                                 └───┘┌─┴─┐┌─────────────┐┌─┴─┐┌──┴───┴───┐»
    «q_3: ─────────────────────────────────┤ X ├┤ U1(0.14302) ├┤ X ├┤ U2(0,pi) ├»
    «                                      └───┘└─────────────┘└───┘└──────────┘»
    «                                  ┌──────────┐            ┌──────────┐     »
    «q_0: ───────────■─────────────────┤ U2(0,pi) ├────────────┤ U2(0,pi) ├─────»
    «              ┌─┴─┐         ┌─────┴──────────┴─────┐┌─────┴──────────┴────┐»
    «q_1: ─────────┤ X ├─────────┤ U3(-pi/2,-pi/2,pi/2) ├┤ U3(pi/2,-pi/2,pi/2) ├»
    «           ┌──┴───┴───┐     ├─────────────────────┬┘└─────────────────────┘»
    «q_2: ──────┤ U2(0,pi) ├─────┤ U3(pi/2,-pi/2,pi/2) ├────────────────────────»
    «     ┌─────┴──────────┴────┐└─────────────────────┘                        »
    «q_3: ┤ U3(pi/2,-pi/2,pi/2) ├───────────────────────────────────────────────»
    «     └─────────────────────┘                                               »
    «                                                                »
    «q_0: ──■────────────────────────────────────────────────────────»
    «     ┌─┴─┐                                                      »
    «q_1: ┤ X ├──■──────────────────────────────────────■────────────»
    «     └───┘┌─┴─┐                                  ┌─┴─┐          »
    «q_2: ─────┤ X ├──■───────────────────■───────────┤ X ├──────────»
    «          └───┘┌─┴─┐┌─────────────┐┌─┴─┐┌────────┴───┴─────────┐»
    «q_3: ──────────┤ X ├┤ U1(0.14302) ├┤ X ├┤ U3(-pi/2,-pi/2,pi/2) ├»
    «               └───┘└─────────────┘└───┘└──────────────────────┘»
    «                                   ┌──────────┐      ┌──────────┐          »
    «q_0: ───────────■──────────────────┤ U2(0,pi) ├──────┤ U2(0,pi) ├──■───────»
    «              ┌─┴─┐          ┌─────┴──────────┴─────┐├──────────┤┌─┴─┐     »
    «q_1: ─────────┤ X ├──────────┤ U3(-pi/2,-pi/2,pi/2) ├┤ U2(0,pi) ├┤ X ├──■──»
    «     ┌────────┴───┴─────────┐├─────────────────────┬┘└──────────┘└───┘┌─┴─┐»
    «q_2: ┤ U3(-pi/2,-pi/2,pi/2) ├┤ U3(pi/2,-pi/2,pi/2) ├──────────────────┤ X ├»
    «     └─────┬──────────┬─────┘└─────────────────────┘                  └───┘»
    «q_3: ──────┤ U2(0,pi) ├────────────────────────────────────────────────────»
    «           └──────────┘                                                    »
    «                                                                   »
    «q_0: ─────────────────────────────────────────────────■────────────»
    «                                                    ┌─┴─┐          »
    «q_1: ───────────────────────────────■───────────────┤ X ├──────────»
    «                                  ┌─┴─┐    ┌────────┴───┴─────────┐»
    «q_2: ──■────────────────────■─────┤ X ├────┤ U3(-pi/2,-pi/2,pi/2) ├»
    «     ┌─┴─┐┌──────────────┐┌─┴─┐┌──┴───┴───┐├─────────────────────┬┘»
    «q_3: ┤ X ├┤ U1(-0.14302) ├┤ X ├┤ U2(0,pi) ├┤ U3(pi/2,-pi/2,pi/2) ├─»
    «     └───┘└──────────────┘└───┘└──────────┘└─────────────────────┘ »
    «     ┌──────────┐┌─────────────────────┐                                   »
    «q_0: ┤ U2(0,pi) ├┤ U3(pi/2,-pi/2,pi/2) ├──■────────────────────────────────»
    «     ├──────────┤├─────────────────────┤┌─┴─┐                              »
    «q_1: ┤ U2(0,pi) ├┤ U3(pi/2,-pi/2,pi/2) ├┤ X ├──■───────────────────────────»
    «     ├──────────┤└─────────────────────┘└───┘┌─┴─┐                         »
    «q_2: ┤ U2(0,pi) ├────────────────────────────┤ X ├──■───────────────────■──»
    «     └──────────┘                            └───┘┌─┴─┐┌─────────────┐┌─┴─┐»
    «q_3: ─────────────────────────────────────────────┤ X ├┤ U1(0.14302) ├┤ X ├»
    «                                                  └───┘└─────────────┘└───┘»
    «                                         ┌──────────────────────┐»
    «q_0: ─────────────────────────────■──────┤ U3(-pi/2,-pi/2,pi/2) ├»
    «                                ┌─┴─┐    ├──────────────────────┤»
    «q_1: ───────────■───────────────┤ X ├────┤ U3(-pi/2,-pi/2,pi/2) ├»
    «              ┌─┴─┐          ┌──┴───┴───┐└─────┬──────────┬─────┘»
    «q_2: ─────────┤ X ├──────────┤ U2(0,pi) ├──────┤ U2(0,pi) ├──────»
    «     ┌────────┴───┴─────────┐├──────────┤      └──────────┘      »
    «q_3: ┤ U3(-pi/2,-pi/2,pi/2) ├┤ U2(0,pi) ├────────────────────────»
    «     └──────────────────────┘└──────────┘                        »
    «     ┌─────────────────────┐                                                »
    «q_0: ┤ U3(pi/2,-pi/2,pi/2) ├──■─────────────────────────────────────────────»
    «     └─────┬──────────┬────┘┌─┴─┐                                           »
    «q_1: ──────┤ U2(0,pi) ├─────┤ X ├──■─────────────────────────────────■──────»
    «           └──────────┘     └───┘┌─┴─┐                             ┌─┴─┐    »
    «q_2: ────────────────────────────┤ X ├──■────────────────────■─────┤ X ├────»
    «                                 └───┘┌─┴─┐┌──────────────┐┌─┴─┐┌──┴───┴───┐»
    «q_3: ─────────────────────────────────┤ X ├┤ U1(-0.14302) ├┤ X ├┤ U2(0,pi) ├»
    «                                      └───┘└──────────────┘└───┘└──────────┘»
    «                            ┌──────────────────────┐┌─────────────────────┐»
    «q_0: ───────────■───────────┤ U3(-pi/2,-pi/2,pi/2) ├┤ U3(pi/2,-pi/2,pi/2) ├»
    «              ┌─┴─┐         └─────┬──────────┬─────┘└─────┬──────────┬────┘»
    «q_1: ─────────┤ X ├───────────────┤ U2(0,pi) ├────────────┤ U2(0,pi) ├─────»
    «           ┌──┴───┴───┐     ┌─────┴──────────┴────┐       └──────────┘     »
    «q_2: ──────┤ U2(0,pi) ├─────┤ U3(pi/2,-pi/2,pi/2) ├────────────────────────»
    «     ┌─────┴──────────┴────┐└─────────────────────┘                        »
    «q_3: ┤ U3(pi/2,-pi/2,pi/2) ├───────────────────────────────────────────────»
    «     └─────────────────────┘                                               »
    «                                                                 »
    «q_0: ──■─────────────────────────────────────────────────────────»
    «     ┌─┴─┐                                                       »
    «q_1: ┤ X ├──■───────────────────────────────────────■────────────»
    «     └───┘┌─┴─┐                                   ┌─┴─┐          »
    «q_2: ─────┤ X ├──■────────────────────■───────────┤ X ├──────────»
    «          └───┘┌─┴─┐┌──────────────┐┌─┴─┐┌────────┴───┴─────────┐»
    «q_3: ──────────┤ X ├┤ U1(-0.14302) ├┤ X ├┤ U3(-pi/2,-pi/2,pi/2) ├»
    «               └───┘└──────────────┘└───┘└──────────────────────┘»
    «                             ┌──────────────────────┐┌─────────────────────┐»
    «q_0: ───────────■────────────┤ U3(-pi/2,-pi/2,pi/2) ├┤ U3(pi/2,-pi/2,pi/2) ├»
    «              ┌─┴─┐          └─────┬──────────┬─────┘├─────────────────────┤»
    «q_1: ─────────┤ X ├────────────────┤ U2(0,pi) ├──────┤ U3(pi/2,-pi/2,pi/2) ├»
    «     ┌────────┴───┴─────────┐┌─────┴──────────┴────┐ └─────────────────────┘»
    «q_2: ┤ U3(-pi/2,-pi/2,pi/2) ├┤ U3(pi/2,-pi/2,pi/2) ├────────────────────────»
    «     └─────┬──────────┬─────┘└─────────────────────┘                        »
    «q_3: ──────┤ U2(0,pi) ├─────────────────────────────────────────────────────»
    «           └──────────┘                                                     »
    «                                                                             »
    «q_0: ──■────────────────────────────────────────────────────────■────────────»
    «     ┌─┴─┐                                                    ┌─┴─┐          »
    «q_1: ┤ X ├──■─────────────────────────────────■───────────────┤ X ├──────────»
    «     └───┘┌─┴─┐                             ┌─┴─┐    ┌────────┴───┴─────────┐»
    «q_2: ─────┤ X ├──■────────────────────■─────┤ X ├────┤ U3(-pi/2,-pi/2,pi/2) ├»
    «          └───┘┌─┴─┐┌──────────────┐┌─┴─┐┌──┴───┴───┐└──────────────────────┘»
    «q_3: ──────────┤ X ├┤ U1(-0.14302) ├┤ X ├┤ U2(0,pi) ├────────────────────────»
    «               └───┘└──────────────┘└───┘└──────────┘                        »
    «     ┌──────────────────────┐
    «q_0: ┤ U3(-pi/2,-pi/2,pi/2) ├
    «     ├──────────────────────┤
    «q_1: ┤ U3(-pi/2,-pi/2,pi/2) ├
    «     └──────────────────────┘
    «q_2: ────────────────────────
    «
    «q_3: ────────────────────────
    «

### Convert the circuit to OpenQASM code

```
print(uccsd_ansatz_circuit.qasm())
```

    OPENQASM 2.0;
    include "qelib1.inc";
    qreg q[4];
    u3(3.14159265358979,0.0,3.14159265358979) q[0];
    u3(3.14159265358979,0.0,3.14159265358979) q[2];
    u2(0.0,pi) q[0];
    u3(pi/2,-pi/2,pi/2) q[1];
    cx q[0],q[1];
    u1(0.849234900739678) q[1];
    cx q[0],q[1];
    u2(0.0,pi) q[0];
    u3(-pi/2,-pi/2,pi/2) q[1];
    u3(pi/2,-pi/2,pi/2) q[0];
    u2(0.0,pi) q[1];
    cx q[0],q[1];
    u1(-0.849234900739678) q[1];
    cx q[0],q[1];
    u3(-pi/2,-pi/2,pi/2) q[0];
    u2(0.0,pi) q[1];
    u2(0.0,pi) q[2];
    u3(pi/2,-pi/2,pi/2) q[3];
    cx q[2],q[3];
    u1(0.284298969229392) q[3];
    cx q[2],q[3];
    u2(0.0,pi) q[2];
    u3(-pi/2,-pi/2,pi/2) q[3];
    u3(pi/2,-pi/2,pi/2) q[2];
    u2(0.0,pi) q[3];
    cx q[2],q[3];
    u1(-0.284298969229392) q[3];
    cx q[2],q[3];
    u3(-pi/2,-pi/2,pi/2) q[2];
    u2(0.0,pi) q[3];
    u2(0.0,pi) q[0];
    u2(0.0,pi) q[1];
    u2(0.0,pi) q[2];
    u3(pi/2,-pi/2,pi/2) q[3];
    cx q[0],q[1];
    cx q[1],q[2];
    cx q[2],q[3];
    u1(0.143019766019107) q[3];
    cx q[2],q[3];
    cx q[1],q[2];
    cx q[0],q[1];
    u2(0.0,pi) q[0];
    u2(0.0,pi) q[1];
    u2(0.0,pi) q[2];
    u3(-pi/2,-pi/2,pi/2) q[3];
    u2(0.0,pi) q[0];
    u3(pi/2,-pi/2,pi/2) q[1];
    u2(0.0,pi) q[2];
    u2(0.0,pi) q[3];
    cx q[0],q[1];
    cx q[1],q[2];
    cx q[2],q[3];
    u1(0.143019766019107) q[3];
    cx q[2],q[3];
    cx q[1],q[2];
    cx q[0],q[1];
    u2(0.0,pi) q[0];
    u3(-pi/2,-pi/2,pi/2) q[1];
    u2(0.0,pi) q[2];
    u2(0.0,pi) q[3];
    u2(0.0,pi) q[0];
    u3(pi/2,-pi/2,pi/2) q[1];
    u3(pi/2,-pi/2,pi/2) q[2];
    u3(pi/2,-pi/2,pi/2) q[3];
    cx q[0],q[1];
    cx q[1],q[2];
    cx q[2],q[3];
    u1(0.143019766019107) q[3];
    cx q[2],q[3];
    cx q[1],q[2];
    cx q[0],q[1];
    u2(0.0,pi) q[0];
    u3(-pi/2,-pi/2,pi/2) q[1];
    u3(-pi/2,-pi/2,pi/2) q[2];
    u3(-pi/2,-pi/2,pi/2) q[3];
    u2(0.0,pi) q[0];
    u2(0.0,pi) q[1];
    u3(pi/2,-pi/2,pi/2) q[2];
    u2(0.0,pi) q[3];
    cx q[0],q[1];
    cx q[1],q[2];
    cx q[2],q[3];
    u1(-0.143019766019107) q[3];
    cx q[2],q[3];
    cx q[1],q[2];
    cx q[0],q[1];
    u2(0.0,pi) q[0];
    u2(0.0,pi) q[1];
    u3(-pi/2,-pi/2,pi/2) q[2];
    u2(0.0,pi) q[3];
    u3(pi/2,-pi/2,pi/2) q[0];
    u3(pi/2,-pi/2,pi/2) q[1];
    u2(0.0,pi) q[2];
    u3(pi/2,-pi/2,pi/2) q[3];
    cx q[0],q[1];
    cx q[1],q[2];
    cx q[2],q[3];
    u1(0.143019766019107) q[3];
    cx q[2],q[3];
    cx q[1],q[2];
    cx q[0],q[1];
    u3(-pi/2,-pi/2,pi/2) q[0];
    u3(-pi/2,-pi/2,pi/2) q[1];
    u2(0.0,pi) q[2];
    u3(-pi/2,-pi/2,pi/2) q[3];
    u3(pi/2,-pi/2,pi/2) q[0];
    u2(0.0,pi) q[1];
    u2(0.0,pi) q[2];
    u2(0.0,pi) q[3];
    cx q[0],q[1];
    cx q[1],q[2];
    cx q[2],q[3];
    u1(-0.143019766019107) q[3];
    cx q[2],q[3];
    cx q[1],q[2];
    cx q[0],q[1];
    u3(-pi/2,-pi/2,pi/2) q[0];
    u2(0.0,pi) q[1];
    u2(0.0,pi) q[2];
    u2(0.0,pi) q[3];
    u3(pi/2,-pi/2,pi/2) q[0];
    u2(0.0,pi) q[1];
    u3(pi/2,-pi/2,pi/2) q[2];
    u3(pi/2,-pi/2,pi/2) q[3];
    cx q[0],q[1];
    cx q[1],q[2];
    cx q[2],q[3];
    u1(-0.143019766019107) q[3];
    cx q[2],q[3];
    cx q[1],q[2];
    cx q[0],q[1];
    u3(-pi/2,-pi/2,pi/2) q[0];
    u2(0.0,pi) q[1];
    u3(-pi/2,-pi/2,pi/2) q[2];
    u3(-pi/2,-pi/2,pi/2) q[3];
    u3(pi/2,-pi/2,pi/2) q[0];
    u3(pi/2,-pi/2,pi/2) q[1];
    u3(pi/2,-pi/2,pi/2) q[2];
    u2(0.0,pi) q[3];
    cx q[0],q[1];
    cx q[1],q[2];
    cx q[2],q[3];
    u1(-0.143019766019107) q[3];
    cx q[2],q[3];
    cx q[1],q[2];
    cx q[0],q[1];
    u3(-pi/2,-pi/2,pi/2) q[0];
    u3(-pi/2,-pi/2,pi/2) q[1];
    u3(-pi/2,-pi/2,pi/2) q[2];
    u2(0.0,pi) q[3];


## Text Mining and NLP

## Part 2

### **Goal**: to internalize the steps, challenges, and methodology of text mining
- explore text analysis by hand
- apply text mining steps in Jupyter with Python libraries NLTK
- classify documents correctly

^ This last step will require modeling!

```python
import nltk
import sklearn
import pandas as pd
import string, re
import urllib

url_a = "https://raw.githubusercontent.com/aapeebles/text_examples/master/Text%20examples%20folder/A.txt"
url_b = "https://raw.githubusercontent.com/aapeebles/text_examples/master/Text%20examples%20folder/D.txt"
article_a = urllib.request.urlopen(url_a).read()
article_a_st = article_a.decode("utf-8")
```

```python
# tokens
pattern = "([a-zA-Z]+(?:'[a-z]+)?)"
arta_tokens_raw = nltk.regexp_tokenize(article_a_st, pattern)

# lower case
arta_tokens = [i.lower() for i in arta_tokens_raw]

# stop words if you get an error here uncommment the next line and run the cell again
# nltk.download('stopwords')
nltk.corpus.stopwords.words("english")

stop_words = set(nltk.corpus.stopwords.words('english'))
arta_tokens_stopped = [w for w in arta_tokens if not w in stop_words]

# stem words
stemmer = nltk.SnowballStemmer("english")
arta_stemmed = [stemmer.stem(word) for word in arta_tokens_stopped]
```

```python
# repeat w second article
article_b = urllib.request.urlopen(url_b).read()
article_b_st = article_b.decode("utf-8")
artb_tokens_raw = nltk.regexp_tokenize(article_b_st, pattern)
artb_tokens = [i.lower() for i in artb_tokens_raw]
artb_tokens_stopped = [w for w in artb_tokens if not w in stop_words]
artb_stemmed = [stemmer.stem(word) for word in artb_tokens_stopped]
```

### Term Frequency (TF)

$\begin{align}
 tf_{i,j} = \dfrac{n_{i,j}}{\displaystyle \sum_k n_{i,j} }
\end{align} $

### Inverse Document Frequency (IDF)

$\begin{align}
idf(w) = \log \dfrac{N}{df_t}
\end{align} $

### TF-IDF score

$ \begin{align}
w_{i,j} = tf_{i,j} \times \log \dfrac{N}{df_i} \\
tf_{i,j} = \text{number of occurences of } i \text{ in} j \\
df_i = \text{number of documents containing } i \\
N = \text{total number of documents}
\end{align} $

#### Counting from scratch

```python
article_b_st
```

    'BNP leader Nick Griffin arrested\n\nThe leader of the British National Party has been arrested as part of a police inquiry following the screening of a BBC documentary.\n\nA party spokesman said Nick Griffin was arrested on Tuesday morning on suspicion of incitement to commit racial hatred. West Yorkshire police confirmed they had arrested a 45-year-old man from outside their area. BNP founding chairman John Tyndall was arrested on Sunday on the same charge.\n\nIn July, the BBC documentary Secret Agent featured covertly-filmed footage of BNP activists. Mr Griffin is the twelfth man to be arrested following the documentary. Nine men from West Yorkshire and another man from Leicester have been arrested and freed on bail. Seven of the men had been held variously in connection with suspected racially aggravated public order offences, conspiracy to commit criminal damage and possession of a firearm. Two men, both from Keighley, were arrested in September on suspicion of conspiracy to commit criminal damage. A 24-year-old man from Leicester was detained on Monday on suspicion of incitement to commit racial hatred. A BNP spokesperson said Mr Tyndall, from Brighton, was arrested following a speech he made in Burnley, Lancashire, and was released on police bail.\n'

```python
wordSet = set(arta_stemmed).union(set(artb_stemmed))
wordDictA = dict.fromkeys(wordSet, 0)
wordDictB = dict.fromkeys(wordSet, 0)

for word in arta_stemmed:
    wordDictA[word]+=1

for word in artb_stemmed:
    wordDictB[word]+=1
```

```python
wordDictA
```

    {'favour': 1,
     'sunday': 0,
     'month': 1,
     'screen': 0,
     'without': 1,
     'chairman': 0,
     'poland': 1,
     'state': 2,
     'brighton': 0,
     'bring': 1,
     'legal': 3,
     'mep': 2,
     'old': 0,
     'aggrav': 0,
     'com': 1,
     'eu': 4,
     'exampl': 1,
     'european': 2,
     'lancashir': 0,
     'vote': 1,
     'effect': 1,
     'fear': 1,
     'seven': 0,
     'direct': 4,
     'reject': 1,
     'parti': 0,
     'method': 1,
     'invent': 5,
     'abstain': 1,
     'comput': 4,
     'gain': 1,
     'oppon': 1,
     'vocal': 1,
     'west': 0,
     'let': 1,
     'leicest': 0,
     'achiev': 1,
     'largest': 1,
     'various': 0,
     'racial': 0,
     'twelfth': 0,
     'court': 1,
     'lead': 1,
     'conspiraci': 0,
     'busi': 1,
     'spokesperson': 0,
     'men': 0,
     'burnley': 0,
     'govern': 1,
     'action': 1,
     'featur': 0,
     'submit': 1,
     'open': 1,
     'intens': 1,
     'concern': 1,
     'secret': 0,
     'freed': 0,
     'speech': 0,
     'griffin': 0,
     'tyndal': 0,
     'impact': 1,
     'chanc': 1,
     'union': 1,
     'critic': 2,
     'held': 0,
     'yorkshir': 0,
     'parliament': 2,
     'put': 1,
     'first': 1,
     'morn': 0,
     'nine': 0,
     'john': 0,
     'servic': 1,
     'year': 0,
     'committe': 2,
     'affair': 1,
     'hurt': 1,
     'happen': 1,
     'support': 2,
     'protect': 2,
     'part': 0,
     'adopt': 1,
     'suspect': 0,
     'member': 2,
     'arrest': 0,
     'firearm': 0,
     'develop': 1,
     'confirm': 0,
     'setback': 1,
     'hold': 1,
     'shop': 1,
     'start': 1,
     'documentari': 0,
     'innov': 1,
     'bnp': 0,
     'permit': 1,
     'twice': 1,
     'man': 0,
     'us': 3,
     'amazon': 1,
     'compani': 1,
     'new': 2,
     'polic': 0,
     'order': 2,
     'rewrit': 1,
     'commiss': 1,
     'larg': 1,
     'bail': 0,
     'bbc': 0,
     'might': 1,
     'back': 2,
     'issu': 1,
     'welcom': 1,
     'say': 3,
     'meet': 1,
     'spokesman': 0,
     'ineffici': 1,
     'program': 1,
     'base': 2,
     'damag': 0,
     'charg': 0,
     'nation': 1,
     'word': 1,
     'debat': 1,
     'small': 2,
     'could': 3,
     'serv': 1,
     'public': 0,
     'outsid': 0,
     'two': 2,
     'offenc': 0,
     'possess': 0,
     'internet': 1,
     'momentum': 1,
     'juli': 0,
     'give': 1,
     'tuesday': 0,
     'use': 1,
     'area': 0,
     'juri': 2,
     'covert': 0,
     'would': 3,
     'read': 1,
     'click': 1,
     'play': 1,
     'fuller': 1,
     'implement': 2,
     'activist': 0,
     'reboot': 1,
     'immens': 1,
     'film': 0,
     'europ': 1,
     'field': 1,
     'law': 5,
     'rule': 1,
     'fight': 1,
     'softwar': 3,
     'mr': 0,
     'british': 0,
     'commit': 0,
     'footag': 0,
     'crimin': 0,
     'suffer': 1,
     'model': 1,
     'larger': 1,
     'keighley': 0,
     'line': 1,
     'offer': 1,
     'suspicion': 0,
     'releas': 0,
     'decis': 1,
     'one': 3,
     'patent': 5,
     'agent': 0,
     'lobbi': 1,
     'anoth': 0,
     'even': 1,
     'similar': 1,
     'draft': 3,
     'sourc': 1,
     'leader': 0,
     'controversi': 1,
     'found': 0,
     'follow': 0,
     'current': 1,
     'fail': 1,
     'financi': 1,
     'intend': 1,
     'nick': 0,
     'made': 0,
     'inquiri': 0,
     'said': 2,
     'propos': 2,
     'connect': 0,
     'implic': 1,
     'firm': 2,
     'pressur': 1,
     'mean': 1,
     'monday': 0,
     'incit': 0,
     'detain': 0,
     'hatr': 0,
     'septemb': 0}

```python
wordDictB
```

    {'favour': 0,
     'sunday': 1,
     'month': 0,
     'screen': 1,
     'without': 0,
     'chairman': 1,
     'poland': 0,
     'state': 0,
     'brighton': 1,
     'bring': 0,
     'legal': 0,
     'mep': 0,
     'old': 2,
     'aggrav': 1,
     'com': 0,
     'eu': 0,
     'exampl': 0,
     'european': 0,
     'lancashir': 1,
     'vote': 0,
     'effect': 0,
     'fear': 0,
     'seven': 1,
     'direct': 0,
     'reject': 0,
     'parti': 2,
     'method': 0,
     'invent': 0,
     'abstain': 0,
     'comput': 0,
     'gain': 0,
     'oppon': 0,
     'vocal': 0,
     'west': 2,
     'let': 0,
     'leicest': 2,
     'achiev': 0,
     'largest': 0,
     'various': 1,
     'racial': 3,
     'twelfth': 1,
     'court': 0,
     'lead': 0,
     'conspiraci': 2,
     'busi': 0,
     'spokesperson': 1,
     'men': 3,
     'burnley': 1,
     'govern': 0,
     'action': 0,
     'featur': 1,
     'submit': 0,
     'open': 0,
     'intens': 0,
     'concern': 0,
     'secret': 1,
     'freed': 1,
     'speech': 1,
     'griffin': 3,
     'tyndal': 2,
     'impact': 0,
     'chanc': 0,
     'union': 0,
     'critic': 0,
     'held': 1,
     'yorkshir': 2,
     'parliament': 0,
     'put': 0,
     'first': 0,
     'morn': 1,
     'nine': 1,
     'john': 1,
     'servic': 0,
     'year': 2,
     'committe': 0,
     'affair': 0,
     'hurt': 0,
     'happen': 0,
     'support': 0,
     'protect': 0,
     'part': 1,
     'adopt': 0,
     'suspect': 1,
     'member': 0,
     'arrest': 9,
     'firearm': 1,
     'develop': 0,
     'confirm': 1,
     'setback': 0,
     'hold': 0,
     'shop': 0,
     'start': 0,
     'documentari': 3,
     'innov': 0,
     'bnp': 4,
     'permit': 0,
     'twice': 0,
     'man': 4,
     'us': 0,
     'amazon': 0,
     'compani': 0,
     'new': 0,
     'polic': 3,
     'order': 1,
     'rewrit': 0,
     'commiss': 0,
     'larg': 0,
     'bail': 2,
     'bbc': 2,
     'might': 0,
     'back': 0,
     'issu': 0,
     'welcom': 0,
     'say': 0,
     'meet': 0,
     'spokesman': 1,
     'ineffici': 0,
     'program': 0,
     'base': 0,
     'damag': 2,
     'charg': 1,
     'nation': 1,
     'word': 0,
     'debat': 0,
     'small': 0,
     'could': 0,
     'serv': 0,
     'public': 1,
     'outsid': 1,
     'two': 1,
     'offenc': 1,
     'possess': 1,
     'internet': 0,
     'momentum': 0,
     'juli': 1,
     'give': 0,
     'tuesday': 1,
     'use': 0,
     'area': 1,
     'juri': 0,
     'covert': 1,
     'would': 0,
     'read': 0,
     'click': 0,
     'play': 0,
     'fuller': 0,
     'implement': 0,
     'activist': 1,
     'reboot': 0,
     'immens': 0,
     'film': 1,
     'europ': 0,
     'field': 0,
     'law': 0,
     'rule': 0,
     'fight': 0,
     'softwar': 0,
     'mr': 2,
     'british': 1,
     'commit': 4,
     'footag': 1,
     'crimin': 2,
     'suffer': 0,
     'model': 0,
     'larger': 0,
     'keighley': 1,
     'line': 0,
     'offer': 0,
     'suspicion': 3,
     'releas': 1,
     'decis': 0,
     'one': 0,
     'patent': 0,
     'agent': 1,
     'lobbi': 0,
     'anoth': 1,
     'even': 0,
     'similar': 0,
     'draft': 0,
     'sourc': 0,
     'leader': 2,
     'controversi': 0,
     'found': 1,
     'follow': 3,
     'current': 0,
     'fail': 0,
     'financi': 0,
     'intend': 0,
     'nick': 2,
     'made': 1,
     'inquiri': 1,
     'said': 2,
     'propos': 0,
     'connect': 1,
     'implic': 0,
     'firm': 0,
     'pressur': 0,
     'mean': 0,
     'monday': 1,
     'incit': 2,
     'detain': 1,
     'hatr': 2,
     'septemb': 1}

```python
df = pd.DataFrame([wordDictA, wordDictB], index=range(2))
df.reindex(sorted(df.columns), axis=1)
```

      abstain
      achiev
      action
      activist
      adopt
      affair
      agent
      aggrav
      amazon
      anoth
      ...
      various
      vocal
      vote
      welcom
      west
      without
      word
      would
      year
      yorkshir

      0
      1
      1
      1
      0
      1
      1
      0
      0
      1
      0
      ...
      0
      1
      1
      1
      0
      1
      1
      3
      0
      0

      1
      0
      0
      0
      1
      0
      0
      1
      1
      0
      1
      ...
      1
      0
      0
      0
      2
      0
      0
      0
      2
      2

2 rows × 203 columns

#### Now with sklearn

```python
counts = sklearn.feature_extraction.text.CountVectorizer()
counts_fitted = counts.fit_transform([article_b_st, article_a_st])

pd.DataFrame(counts_fitted.toarray(), columns=counts.get_feature_names())
```

      24
      45
      about
      abstain
      achieve
      action
      activists
      adoption
      affairs
      after
      ...
      west
      when
      which
      who
      with
      without
      words
      would
      year
      yorkshire

      0
      1
      1
      0
      0
      0
      0
      1
      0
      0
      0
      ...
      2
      0
      0
      0
      1
      0
      0
      0
      2
      2

      1
      0
      0
      1
      1
      1
      1
      0
      1
      1
      1
      ...
      0
      1
      1
      1
      2
      1
      1
      3
      0
      0

2 rows × 261 columns

## Tfidf from scratch

```python
def computeTF(wordDict, bow):
    tfDict = {}
    bowCount = len(bow)
    for word, count in wordDict.items():
        tfDict[word] = count / bowCount
    return tfDict
```

```python
tfbowA = computeTF(wordDictA, arta_stemmed)
tfbowB = computeTF(wordDictB, artb_stemmed)
```

```python
tfbowA
```

    {'favour': 0.005434782608695652,
     'sunday': 0.0,
     'month': 0.005434782608695652,
     'screen': 0.0,
     'without': 0.005434782608695652,
     'chairman': 0.0,
     'poland': 0.005434782608695652,
     'state': 0.010869565217391304,
     'brighton': 0.0,
     'bring': 0.005434782608695652,
     'legal': 0.016304347826086956,
     'mep': 0.010869565217391304,
     'old': 0.0,
     'aggrav': 0.0,
     'com': 0.005434782608695652,
     'eu': 0.021739130434782608,
     'exampl': 0.005434782608695652,
     'european': 0.010869565217391304,
     'lancashir': 0.0,
     'vote': 0.005434782608695652,
     'effect': 0.005434782608695652,
     'fear': 0.005434782608695652,
     'seven': 0.0,
     'direct': 0.021739130434782608,
     'reject': 0.005434782608695652,
     'parti': 0.0,
     'method': 0.005434782608695652,
     'invent': 0.02717391304347826,
     'abstain': 0.005434782608695652,
     'comput': 0.021739130434782608,
     'gain': 0.005434782608695652,
     'oppon': 0.005434782608695652,
     'vocal': 0.005434782608695652,
     'west': 0.0,
     'let': 0.005434782608695652,
     'leicest': 0.0,
     'achiev': 0.005434782608695652,
     'largest': 0.005434782608695652,
     'various': 0.0,
     'racial': 0.0,
     'twelfth': 0.0,
     'court': 0.005434782608695652,
     'lead': 0.005434782608695652,
     'conspiraci': 0.0,
     'busi': 0.005434782608695652,
     'spokesperson': 0.0,
     'men': 0.0,
     'burnley': 0.0,
     'govern': 0.005434782608695652,
     'action': 0.005434782608695652,
     'featur': 0.0,
     'submit': 0.005434782608695652,
     'open': 0.005434782608695652,
     'intens': 0.005434782608695652,
     'concern': 0.005434782608695652,
     'secret': 0.0,
     'freed': 0.0,
     'speech': 0.0,
     'griffin': 0.0,
     'tyndal': 0.0,
     'impact': 0.005434782608695652,
     'chanc': 0.005434782608695652,
     'union': 0.005434782608695652,
     'critic': 0.010869565217391304,
     'held': 0.0,
     'yorkshir': 0.0,
     'parliament': 0.010869565217391304,
     'put': 0.005434782608695652,
     'first': 0.005434782608695652,
     'morn': 0.0,
     'nine': 0.0,
     'john': 0.0,
     'servic': 0.005434782608695652,
     'year': 0.0,
     'committe': 0.010869565217391304,
     'affair': 0.005434782608695652,
     'hurt': 0.005434782608695652,
     'happen': 0.005434782608695652,
     'support': 0.010869565217391304,
     'protect': 0.010869565217391304,
     'part': 0.0,
     'adopt': 0.005434782608695652,
     'suspect': 0.0,
     'member': 0.010869565217391304,
     'arrest': 0.0,
     'firearm': 0.0,
     'develop': 0.005434782608695652,
     'confirm': 0.0,
     'setback': 0.005434782608695652,
     'hold': 0.005434782608695652,
     'shop': 0.005434782608695652,
     'start': 0.005434782608695652,
     'documentari': 0.0,
     'innov': 0.005434782608695652,
     'bnp': 0.0,
     'permit': 0.005434782608695652,
     'twice': 0.005434782608695652,
     'man': 0.0,
     'us': 0.016304347826086956,
     'amazon': 0.005434782608695652,
     'compani': 0.005434782608695652,
     'new': 0.010869565217391304,
     'polic': 0.0,
     'order': 0.010869565217391304,
     'rewrit': 0.005434782608695652,
     'commiss': 0.005434782608695652,
     'larg': 0.005434782608695652,
     'bail': 0.0,
     'bbc': 0.0,
     'might': 0.005434782608695652,
     'back': 0.010869565217391304,
     'issu': 0.005434782608695652,
     'welcom': 0.005434782608695652,
     'say': 0.016304347826086956,
     'meet': 0.005434782608695652,
     'spokesman': 0.0,
     'ineffici': 0.005434782608695652,
     'program': 0.005434782608695652,
     'base': 0.010869565217391304,
     'damag': 0.0,
     'charg': 0.0,
     'nation': 0.005434782608695652,
     'word': 0.005434782608695652,
     'debat': 0.005434782608695652,
     'small': 0.010869565217391304,
     'could': 0.016304347826086956,
     'serv': 0.005434782608695652,
     'public': 0.0,
     'outsid': 0.0,
     'two': 0.010869565217391304,
     'offenc': 0.0,
     'possess': 0.0,
     'internet': 0.005434782608695652,
     'momentum': 0.005434782608695652,
     'juli': 0.0,
     'give': 0.005434782608695652,
     'tuesday': 0.0,
     'use': 0.005434782608695652,
     'area': 0.0,
     'juri': 0.010869565217391304,
     'covert': 0.0,
     'would': 0.016304347826086956,
     'read': 0.005434782608695652,
     'click': 0.005434782608695652,
     'play': 0.005434782608695652,
     'fuller': 0.005434782608695652,
     'implement': 0.010869565217391304,
     'activist': 0.0,
     'reboot': 0.005434782608695652,
     'immens': 0.005434782608695652,
     'film': 0.0,
     'europ': 0.005434782608695652,
     'field': 0.005434782608695652,
     'law': 0.02717391304347826,
     'rule': 0.005434782608695652,
     'fight': 0.005434782608695652,
     'softwar': 0.016304347826086956,
     'mr': 0.0,
     'british': 0.0,
     'commit': 0.0,
     'footag': 0.0,
     'crimin': 0.0,
     'suffer': 0.005434782608695652,
     'model': 0.005434782608695652,
     'larger': 0.005434782608695652,
     'keighley': 0.0,
     'line': 0.005434782608695652,
     'offer': 0.005434782608695652,
     'suspicion': 0.0,
     'releas': 0.0,
     'decis': 0.005434782608695652,
     'one': 0.016304347826086956,
     'patent': 0.02717391304347826,
     'agent': 0.0,
     'lobbi': 0.005434782608695652,
     'anoth': 0.0,
     'even': 0.005434782608695652,
     'similar': 0.005434782608695652,
     'draft': 0.016304347826086956,
     'sourc': 0.005434782608695652,
     'leader': 0.0,
     'controversi': 0.005434782608695652,
     'found': 0.0,
     'follow': 0.0,
     'current': 0.005434782608695652,
     'fail': 0.005434782608695652,
     'financi': 0.005434782608695652,
     'intend': 0.005434782608695652,
     'nick': 0.0,
     'made': 0.0,
     'inquiri': 0.0,
     'said': 0.010869565217391304,
     'propos': 0.010869565217391304,
     'connect': 0.0,
     'implic': 0.005434782608695652,
     'firm': 0.010869565217391304,
     'pressur': 0.005434782608695652,
     'mean': 0.005434782608695652,
     'monday': 0.0,
     'incit': 0.0,
     'detain': 0.0,
     'hatr': 0.0,
     'septemb': 0.0}

```python
def computeIDF(docList):
    """ compute inverse doc freq for each doc in the docList
    returns: IDF for each doc
    """
    import math
    idfDict = {}
    N = len(docList)

    idfDict = dict.fromkeys(docList[0].keys(), 0)
    for doc in docList:
        for word, val in doc.items():
            if val > 0:
                idfDict[word] += 1

    for word, val in idfDict.items():
        idfDict[word] = math.log(N / val)

    return idfDict
```

```python
idfs = computeIDF([wordDictA, wordDictB])
```

```python
idfs
```

    {'favour': 0.6931471805599453,
     'sunday': 0.6931471805599453,
     'month': 0.6931471805599453,
     'screen': 0.6931471805599453,
     'without': 0.6931471805599453,
     'chairman': 0.6931471805599453,
     'poland': 0.6931471805599453,
     'state': 0.6931471805599453,
     'brighton': 0.6931471805599453,
     'bring': 0.6931471805599453,
     'legal': 0.6931471805599453,
     'mep': 0.6931471805599453,
     'old': 0.6931471805599453,
     'aggrav': 0.6931471805599453,
     'com': 0.6931471805599453,
     'eu': 0.6931471805599453,
     'exampl': 0.6931471805599453,
     'european': 0.6931471805599453,
     'lancashir': 0.6931471805599453,
     'vote': 0.6931471805599453,
     'effect': 0.6931471805599453,
     'fear': 0.6931471805599453,
     'seven': 0.6931471805599453,
     'direct': 0.6931471805599453,
     'reject': 0.6931471805599453,
     'parti': 0.6931471805599453,
     'method': 0.6931471805599453,
     'invent': 0.6931471805599453,
     'abstain': 0.6931471805599453,
     'comput': 0.6931471805599453,
     'gain': 0.6931471805599453,
     'oppon': 0.6931471805599453,
     'vocal': 0.6931471805599453,
     'west': 0.6931471805599453,
     'let': 0.6931471805599453,
     'leicest': 0.6931471805599453,
     'achiev': 0.6931471805599453,
     'largest': 0.6931471805599453,
     'various': 0.6931471805599453,
     'racial': 0.6931471805599453,
     'twelfth': 0.6931471805599453,
     'court': 0.6931471805599453,
     'lead': 0.6931471805599453,
     'conspiraci': 0.6931471805599453,
     'busi': 0.6931471805599453,
     'spokesperson': 0.6931471805599453,
     'men': 0.6931471805599453,
     'burnley': 0.6931471805599453,
     'govern': 0.6931471805599453,
     'action': 0.6931471805599453,
     'featur': 0.6931471805599453,
     'submit': 0.6931471805599453,
     'open': 0.6931471805599453,
     'intens': 0.6931471805599453,
     'concern': 0.6931471805599453,
     'secret': 0.6931471805599453,
     'freed': 0.6931471805599453,
     'speech': 0.6931471805599453,
     'griffin': 0.6931471805599453,
     'tyndal': 0.6931471805599453,
     'impact': 0.6931471805599453,
     'chanc': 0.6931471805599453,
     'union': 0.6931471805599453,
     'critic': 0.6931471805599453,
     'held': 0.6931471805599453,
     'yorkshir': 0.6931471805599453,
     'parliament': 0.6931471805599453,
     'put': 0.6931471805599453,
     'first': 0.6931471805599453,
     'morn': 0.6931471805599453,
     'nine': 0.6931471805599453,
     'john': 0.6931471805599453,
     'servic': 0.6931471805599453,
     'year': 0.6931471805599453,
     'committe': 0.6931471805599453,
     'affair': 0.6931471805599453,
     'hurt': 0.6931471805599453,
     'happen': 0.6931471805599453,
     'support': 0.6931471805599453,
     'protect': 0.6931471805599453,
     'part': 0.6931471805599453,
     'adopt': 0.6931471805599453,
     'suspect': 0.6931471805599453,
     'member': 0.6931471805599453,
     'arrest': 0.6931471805599453,
     'firearm': 0.6931471805599453,
     'develop': 0.6931471805599453,
     'confirm': 0.6931471805599453,
     'setback': 0.6931471805599453,
     'hold': 0.6931471805599453,
     'shop': 0.6931471805599453,
     'start': 0.6931471805599453,
     'documentari': 0.6931471805599453,
     'innov': 0.6931471805599453,
     'bnp': 0.6931471805599453,
     'permit': 0.6931471805599453,
     'twice': 0.6931471805599453,
     'man': 0.6931471805599453,
     'us': 0.6931471805599453,
     'amazon': 0.6931471805599453,
     'compani': 0.6931471805599453,
     'new': 0.6931471805599453,
     'polic': 0.6931471805599453,
     'order': 0.0,
     'rewrit': 0.6931471805599453,
     'commiss': 0.6931471805599453,
     'larg': 0.6931471805599453,
     'bail': 0.6931471805599453,
     'bbc': 0.6931471805599453,
     'might': 0.6931471805599453,
     'back': 0.6931471805599453,
     'issu': 0.6931471805599453,
     'welcom': 0.6931471805599453,
     'say': 0.6931471805599453,
     'meet': 0.6931471805599453,
     'spokesman': 0.6931471805599453,
     'ineffici': 0.6931471805599453,
     'program': 0.6931471805599453,
     'base': 0.6931471805599453,
     'damag': 0.6931471805599453,
     'charg': 0.6931471805599453,
     'nation': 0.0,
     'word': 0.6931471805599453,
     'debat': 0.6931471805599453,
     'small': 0.6931471805599453,
     'could': 0.6931471805599453,
     'serv': 0.6931471805599453,
     'public': 0.6931471805599453,
     'outsid': 0.6931471805599453,
     'two': 0.0,
     'offenc': 0.6931471805599453,
     'possess': 0.6931471805599453,
     'internet': 0.6931471805599453,
     'momentum': 0.6931471805599453,
     'juli': 0.6931471805599453,
     'give': 0.6931471805599453,
     'tuesday': 0.6931471805599453,
     'use': 0.6931471805599453,
     'area': 0.6931471805599453,
     'juri': 0.6931471805599453,
     'covert': 0.6931471805599453,
     'would': 0.6931471805599453,
     'read': 0.6931471805599453,
     'click': 0.6931471805599453,
     'play': 0.6931471805599453,
     'fuller': 0.6931471805599453,
     'implement': 0.6931471805599453,
     'activist': 0.6931471805599453,
     'reboot': 0.6931471805599453,
     'immens': 0.6931471805599453,
     'film': 0.6931471805599453,
     'europ': 0.6931471805599453,
     'field': 0.6931471805599453,
     'law': 0.6931471805599453,
     'rule': 0.6931471805599453,
     'fight': 0.6931471805599453,
     'softwar': 0.6931471805599453,
     'mr': 0.6931471805599453,
     'british': 0.6931471805599453,
     'commit': 0.6931471805599453,
     'footag': 0.6931471805599453,
     'crimin': 0.6931471805599453,
     'suffer': 0.6931471805599453,
     'model': 0.6931471805599453,
     'larger': 0.6931471805599453,
     'keighley': 0.6931471805599453,
     'line': 0.6931471805599453,
     'offer': 0.6931471805599453,
     'suspicion': 0.6931471805599453,
     'releas': 0.6931471805599453,
     'decis': 0.6931471805599453,
     'one': 0.6931471805599453,
     'patent': 0.6931471805599453,
     'agent': 0.6931471805599453,
     'lobbi': 0.6931471805599453,
     'anoth': 0.6931471805599453,
     'even': 0.6931471805599453,
     'similar': 0.6931471805599453,
     'draft': 0.6931471805599453,
     'sourc': 0.6931471805599453,
     'leader': 0.6931471805599453,
     'controversi': 0.6931471805599453,
     'found': 0.6931471805599453,
     'follow': 0.6931471805599453,
     'current': 0.6931471805599453,
     'fail': 0.6931471805599453,
     'financi': 0.6931471805599453,
     'intend': 0.6931471805599453,
     'nick': 0.6931471805599453,
     'made': 0.6931471805599453,
     'inquiri': 0.6931471805599453,
     'said': 0.0,
     'propos': 0.6931471805599453,
     'connect': 0.6931471805599453,
     'implic': 0.6931471805599453,
     'firm': 0.6931471805599453,
     'pressur': 0.6931471805599453,
     'mean': 0.6931471805599453,
     'monday': 0.6931471805599453,
     'incit': 0.6931471805599453,
     'detain': 0.6931471805599453,
     'hatr': 0.6931471805599453,
     'septemb': 0.6931471805599453}

```python
def computeTFIDF(tfBow, idfs):
    """creates function for computing TFIDF"""
    tfidf = {} # creates empty dictionary
    for word, val in tfBow.items(): #starts a for loop using keys (word) and values from tfBow
        tfidf[word] = val * idfs[word] #for each word in tfBow, the value is multiplied by the idfs for the word.
                                        #The word and resulting computation are then added to the dictionary tfidf
    return tfidf #returns the dictionary tfidf
```

```python
tfidfBowA = computeTFIDF(tfbowA, idfs)
tfidfBowB = computeTFIDF(tfbowB, idfs)
```

```python
import pandas as pd
nlpbh = pd.DataFrame([tfidfBowA, tfidfBowB])
nlpbh
```

      abstain
      achiev
      action
      activist
      adopt
      affair
      agent
      aggrav
      amazon
      anoth
      ...
      various
      vocal
      vote
      welcom
      west
      without
      word
      would
      year
      yorkshir

      0
      0.003767
      0.003767
      0.003767
      0.000000
      0.003767
      0.003767
      0.000000
      0.000000
      0.003767
      0.000000
      ...
      0.000000
      0.003767
      0.003767
      0.003767
      0.000000
      0.003767
      0.003767
      0.011301
      0.000000
      0.000000

      1
      0.000000
      0.000000
      0.000000
      0.005373
      0.000000
      0.000000
      0.005373
      0.005373
      0.000000
      0.005373
      ...
      0.005373
      0.000000
      0.000000
      0.000000
      0.010746
      0.000000
      0.000000
      0.000000
      0.010746
      0.010746

2 rows × 203 columns

Now with sklearn!

```python
def tokenize(text):
    tokens = nltk.word_tokenize(text)
    stemmer = nltk.SnowballStemmer("english")
    stems = []
    for item in tokens:
        stems.append(stemmer.stem(item))
    return stems

tfidf = sklearn.feature_extraction.text.TfidfVectorizer(stop_words='english', tokenizer=tokenize)

response = tfidf.fit_transform([article_b_st, article_a_st])

nlpskl = pd.DataFrame(response.toarray(), columns=tfidf.get_feature_names())
nlpskl
```

      ''
      (
      )
      ,
      .
      24-year-old
      45-year-old
      ``
      abstain
      achiev
      ...
      union
      us-bas
      use
      various
      vocal
      vote
      welcom
      west
      word
      yorkshir

      0
      0.000000
      0.000000
      0.000000
      0.286874
      0.394452
      0.050399
      0.050399
      0.000000
      0.000000
      0.000000
      ...
      0.000000
      0.000000
      0.000000
      0.050399
      0.000000
      0.000000
      0.000000
      0.100798
      0.000000
      0.100798

      1
      0.093784
      0.046892
      0.046892
      0.300275
      0.467094
      0.000000
      0.000000
      0.093784
      0.046892
      0.046892
      ...
      0.046892
      0.046892
      0.046892
      0.000000
      0.046892
      0.046892
      0.046892
      0.000000
      0.046892
      0.000000

2 rows × 193 columns

**Note that these values are different! Sklearn's formula for tf-idf is a little more sophisticated than ours. See the doc for the transformer [here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html).**

### N-grams

Notice that the `TfidfVectorizer()` has a parameter called "ngram_range". Sometimes we want to search not only for individual words but for pairs or triples (etc.) of words. Using $N$ as a variable for the size of the word cluster to consider, we speak of "N-grams". Notice that our default is (1, 1).

### Cosine Similarity

```python
sklearn.metrics.pairwise.cosine_similarity(response)
```

    array([[1.       , 0.2787621],
           [0.2787621, 1.       ]])

### Naive Bayes Modeling

Naive Bayes models lend themselves well to NLP problems. Consider the task of trying to predict genre from text. My subjective probability that a text belongs to a certain genre would be a function of the words in the text. So e.g. the (prior) probability that a text is science-fiction may be relatively small. But the probability that a text is science-fiction *given that it uses the word 'cyclotron'* may be quite high.

Now: What's "naive" about Naive Bayes models?

The calculation of the relevant probabilities could get very complicated. But they get much simpler with the (relatively implausible!) assumption that the different features (occurrences of particular words, in our present case of NLP) are *independent*.

```python
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
```

```python
fetch_20newsgroups().target_names
```

```python
train = fetch_20newsgroups(subset='train')
test = fetch_20newsgroups(subset='test')
```

```python
count = CountVectorizer(stop_words='english', tokenizer=tokenize)
```

```python
X_train = count.fit_transform(train.data)
X_test = count.transform(test.data)
y_train = train.target
y_test = test.target
```

```python
y_test[:10]
```

Which NB model do you want? Check out the options and their differences [here](https://scikit-learn.org/stable/modules/naive_bayes.html).

```python
nb = MultinomialNB()
nb.fit(X_train, y_train)
nb.score(X_test, y_test)
```

```python
from sklearn.metrics import confusion_matrix
```

```python
confusion_matrix(y_test, nb.predict(X_test))
```

## Corpus Statistics

How many non-zero elements are there?

```python
import numpy as np
https://www.gutenberg.org/ebooks/7849.txt.utf-8
newval = np.array(df)

non_zero_vals = np.count_nonzero(newval) / float(df.shape[0])
print(f'Average Number of Non-Zero Elements in Vectorized Articles: {non_zero_vals}')

percent_sparse = len([col for col in df.columns if sum(df[col]) <= 1]) / df.shape[1]
print(f'Percentage of columns containing 0: {percent_sparse}')
```

### Next Steps:
- Create the tf-idf for the **whole** corpus of 12 articles
- What are _on average_ the most important words in the whole corpus?
- Add a column named "Target" to the dataset
- Target will be set to 1 or 0 if the article is "Politics" or "Not Politics"
- Do some exploratory analysis of the dataset
 - What are the average most important words for the "Politics" articles?
 - What are the average most important words for the "Not Politics"?

```python

```

## Lets talk classification
- How would you split into train and test? what would be the dataset?

```python
# Sample code
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
```

```python
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
import nltk
import requests
from random import randint
```

## Download Frankenstein and Metamorphosis

```python
frankenstein = requests.get('https://www.gutenberg.org/files/84/84-0.txt')
frankenstein.encoding ='utf-8'
frankenstein = frankenstein.text

metamorphosis = requests.get('https://www.gutenberg.org/ebooks/5200.txt.utf-8')
metamorphosis.encoding ='utf-8'
metamorphosis = metamorphosis.text

trial = requests.get('https://www.gutenberg.org/ebooks/7849.txt.utf-8')
trial.encoding ='utf-8'
trial = trial.text
```

### Remove some extraneous characters

```python
frank = (frankenstein
         .replace('\r', '')
         .replace('\n', ' ')
         .strip()[1000:]
         .split()
        )

meta = (metamorphosis
        .replace('\r', '')
        .replace('\n', ' ')
        .strip()[1000:]
        .split()
       )

tri = (trial
        .replace('\r', '')
        .replace('\n', ' ')
        .strip()[1000:]
        .split()
       )
```

## Create our dataset by making our observations random subsets from each book

```python
def text_sample(book, sample_length=10):
    start = randint(0, len(meta) - sample_length)
    end = start + sample_length

    return ' '.join(book[start:end])

n_samples = 100
all_text = []

all_text.extend([{'text': text_sample(tri), 'label': 'The Trial'} for _ in range(n_samples)])
all_text.extend([{'text': text_sample(meta), 'label': 'Metamorphosis'} for _ in range(n_samples)])

data = pd.DataFrame(all_text)
```

```python
data.sample(1)['text'].values
```

    array(['neglectful with him or especially indifferent, and he decided to'],
          dtype=object)

```python
X_train, X_test, y_train, y_test = train_test_split(data['text'], data['label'])
```

```python
c = CountVectorizer()

X_train_vec = c.fit_transform(X_train)
X_test_vec = c.transform(X_test)
```

```python
from sklearn.naive_bayes import MultinomialNB

nb = MultinomialNB()

nb.fit(X_train_vec, y_train)

nb.score(X_test_vec, y_test)
```

    0.7

```python

```


# Introduction to Python Pre-Processing

## Libraries

Libraries are like your trusted brands. They provide you with the best of the best tools (for free)

We'll be using
- **pandas**
- **numpy**

*Rule of Thumb: pandas for 2D things (like Excel), numpy for nD things (for more general cases)*

A specific package for your project is better than a package that deals with a wide array of things!

## Dataset (telecom_churn.csv)

The data we'll be using will be bundled together with these files, feel free to look at it in notepad

## Translating Thoughts Into Programming

In my years of programming, I always turn to my trusty tool, that is English. I'll show you how to program by googling

## Viewing the Data

The very first step would be to look at the data.

Google: python load csv pandas

```python
import pandas

df = pandas.read_csv("telecom_churn.csv")
```

As **convention**, users usually indicate loaded pandas **DataFrame** as df, or anything similar. This is up to you though!

There's also a way to view the data

You can *technically* print it but...

```python
# print(df)
```

That looks horrible! Can we do better?

What you want is to get a taste of the data,

thus, you get the first few rows, the **head** of the data

```python
df.head()
```

      State
      Account length
      Area code
      International plan
      Voice mail plan
      Number vmail messages
      Total day minutes
      Total day calls
      Total day charge
      Total eve minutes
      Total eve calls
      Total eve charge
      Total night minutes
      Total night calls
      Total night charge
      Total intl minutes
      Total intl calls
      Total intl charge
      Customer service calls
      Churn

      0
      KS
      128
      415
      No
      Yes
      25
      265.1
      110
      45.07
      197.4
      99
      16.78
      244.7
      91
      11.01
      10.0
      3
      2.70
      1
      False

      1
      OH
      107
      415
      No
      Yes
      26
      161.6
      123
      27.47
      195.5
      103
      16.62
      254.4
      103
      11.45
      13.7
      3
      3.70
      1
      False

      2
      NJ
      137
      415
      No
      No
      0
      243.4
      114
      41.38
      121.2
      110
      10.30
      162.6
      104
      7.32
      12.2
      5
      3.29
      0
      False

      3
      OH
      84
      408
      Yes
      No
      0
      299.4
      71
      50.90
      61.9
      88
      5.26
      196.9
      89
      8.86
      6.6
      7
      1.78
      2
      False

      4
      OK
      75
      415
      Yes
      No
      0
      166.7
      113
      28.34
      148.3
      122
      12.61
      186.9
      121
      8.41
      10.1
      3
      2.73
      3
      False

## Selecting

Let's say we want to isolate, remove some columns/rows

We can do those here!

### Column

```python
df['State'].head()
```

    0    KS
    1    OH
    2    NJ
    3    OH
    4    OK
    Name: State, dtype: object

### Row

```python
df.iloc[1].head()
```

    State                  OH
    Account length        107
    Area code             415
    International plan     No
    Voice mail plan       Yes
    Name: 1, dtype: object

## Tutorial 3.1

Chris from Dep A: ```Get the states and area codes of the file, we also want to know if the customer has churned or not. Can you also export it as out.csv and send it to me?```

\begin{align}
Double\, Click\, To\, Reveal\, Tip\, (1)
%Seems like you need to select multiple columns, Google pandas select multiple columns
\end{align}

\begin{align}
Double\, Click\, To\, Reveal\, Tip\, (2)
%You need to export it as a csv, google pandas to csv
\end{align}

\begin{align}
Double\, Click\, To\, Reveal\, Answer
%df_filter = df[['State', 'Area code', 'Churn']]
%df_filter.to_csv("out.csv", index=False); # index False prevents python from printing the row numbers
%df_test = pandas.read_csv("out.csv")
%df_test.head()
\end{align}

```python
# Your Code here

# Test if you exported it correctly!
# df_test = pandas.read_csv("out.csv")
# df_test.head()
```

      State
      Area code
      Churn

      0
      KS
      415
      False

      1
      OH
      415
      False

      2
      NJ
      415
      False

      3
      OH
      408
      False

      4
      OK
      415
      False

## Splitting

Splitting is quite common if you need to split a string.

The current DataFrame doesn't require splitting, so I'll create one that requires splitting

```python
df_a = \
    pandas.DataFrame({
        'NAME_AGE':['JIN_33', 'ALBERT_51', 'KURT_13', 'ZEWEN_9']
    })
```

```python
df_a
```

      NAME_AGE

      0
      JIN_33

      1
      ALBERT_51

      2
      KURT_13

      3
      ZEWEN_9

\begin{align}
Double\, Click\, To\, Reveal\, Tip\, (1)
%You can google this too!
%Here's an example: dataframe["column_to_split"].str.split("_", n = 1, expand = True)
\end{align}

\begin{align}
Double\, Click\, To\, Reveal\, Tip\, (1)
%df_a["NAME_AGE"].str.split("_", n = 1, expand = True)
\end{align}

```python

```

.ipynb_checkpoints/Intro To Python Data Processing-checkpoint.ipynb