https://www.cs.cornell.edu/projects/kddcup/datasets.html
!tar xzvf hep-th-2003.tar.gz!ls -hal 2003/ | wc -l
# load libraries

```python
# https://github.com/alvinwan/tex2py
!pip install tex2py
```

    Collecting tex2py
    Collecting pytest (from tex2py)
      Using cached https://files.pythonhosted.org/packages/ca/e1/2f229554e5c273962fae8b286395d5bbcc7bef276d2b40e1bad954993db2/pytest-5.1.3-py3-none-any.whl
    Collecting coverage==3.7.1 (from tex2py)
    Collecting TexSoup==0.1.4 (from tex2py)
    Collecting coveralls==1.1 (from tex2py)
      Using cached https://files.pythonhosted.org/packages/81/f9/da2ad9334f60e418cb229c58724330be2635b369676d3eec8f61e1f85faa/coveralls-1.1-py2.py3-none-any.whl
    Collecting pytest-cov==2.5.1 (from tex2py)
      Using cached https://files.pythonhosted.org/packages/30/7d/7f6a78ae44a1248ee28cc777586c18b28a1df903470e5d34a6e25712b8aa/pytest_cov-2.5.1-py2.py3-none-any.whl
    Collecting pptree==2.0 (from tex2py)
    Collecting atomicwrites>=1.0 (from pytest->tex2py)
      Using cached https://files.pythonhosted.org/packages/52/90/6155aa926f43f2b2a22b01be7241be3bfd1ceaf7d0b3267213e8127d41f4/atomicwrites-1.3.0-py2.py3-none-any.whl
    Collecting more-itertools>=4.0.0 (from pytest->tex2py)
      Using cached https://files.pythonhosted.org/packages/45/dc/3241eef99eb45f1def35cf93af35d1cf9ef4c0991792583b8f33ea41b092/more_itertools-7.2.0-py3-none-any.whl
    Collecting pluggy<1.0,>=0.12 (from pytest->tex2py)
      Using cached https://files.pythonhosted.org/packages/92/c7/48439f7d5fd6bddb4c04b850bb862b42e3e2b98570040dfaf68aedd8114b/pluggy-0.13.0-py2.py3-none-any.whl
    Collecting py>=1.5.0 (from pytest->tex2py)
      Using cached https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl
    Collecting attrs>=17.4.0 (from pytest->tex2py)
      Using cached https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
    Collecting importlib-metadata>=0.12; python_version < "3.8" (from pytest->tex2py)
      Using cached https://files.pythonhosted.org/packages/f6/d2/40b3fa882147719744e6aa50ac39cf7a22a913cbcba86a0371176c425a3b/importlib_metadata-0.23-py2.py3-none-any.whl
    Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from pytest->tex2py) (18.0)
    Requirement already satisfied: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest->tex2py) (0.1.7)
    Requirement already satisfied: requests>=1.0.0 in /opt/conda/lib/python3.6/site-packages (from coveralls==1.1->tex2py) (2.20.0)
    Collecting docopt>=0.6.1 (from coveralls==1.1->tex2py)
    Collecting zipp>=0.5 (from importlib-metadata>=0.12; python_version < "3.8"->pytest->tex2py)
      Using cached https://files.pythonhosted.org/packages/74/3d/1ee25a26411ba0401b43c6376d2316a71addcc72ef8690b101b4ea56d76a/zipp-0.6.0-py2.py3-none-any.whl
    Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->tex2py) (1.11.0)
    Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->tex2py) (2.2.2)
    Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests>=1.0.0->coveralls==1.1->tex2py) (1.23)
    Requirement already satisfied: idna<2.8,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=1.0.0->coveralls==1.1->tex2py) (2.7)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=1.0.0->coveralls==1.1->tex2py) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests>=1.0.0->coveralls==1.1->tex2py) (2018.11.29)
    Installing collected packages: atomicwrites, more-itertools, zipp, importlib-metadata, pluggy, py, attrs, pytest, coverage, docopt, coveralls, pytest-cov, TexSoup, pptree, tex2py
    Successfully installed TexSoup-0.1.4 atomicwrites-1.3.0 attrs-19.1.0 coverage-3.7.1 coveralls-1.1 docopt-0.6.2 importlib-metadata-0.23 more-itertools-7.2.0 pluggy-0.13.0 pptree-2.0 py-1.8.0 pytest-5.1.3 pytest-cov-2.5.1 tex2py-0.0.5 zipp-0.6.0

```python
# https://github.com/alvinwan/TexSoup
!pip install texsoup
```

    Requirement already satisfied: texsoup in /opt/conda/lib/python3.6/site-packages (0.1.4)
    Requirement already satisfied: pytest in /opt/conda/lib/python3.6/site-packages (from texsoup) (5.1.3)
    Requirement already satisfied: pytest-cov==2.5.1 in /opt/conda/lib/python3.6/site-packages (from texsoup) (2.5.1)
    Requirement already satisfied: coveralls==1.1 in /opt/conda/lib/python3.6/site-packages (from texsoup) (1.1)
    Requirement already satisfied: coverage==3.7.1 in /opt/conda/lib/python3.6/site-packages (from texsoup) (3.7.1)
    Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from pytest->texsoup) (18.0)
    Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from pytest->texsoup) (19.1.0)
    Requirement already satisfied: atomicwrites>=1.0 in /opt/conda/lib/python3.6/site-packages (from pytest->texsoup) (1.3.0)
    Requirement already satisfied: more-itertools>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from pytest->texsoup) (7.2.0)
    Requirement already satisfied: pluggy<1.0,>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->texsoup) (0.13.0)
    Requirement already satisfied: py>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from pytest->texsoup) (1.8.0)
    Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in /opt/conda/lib/python3.6/site-packages (from pytest->texsoup) (0.23)
    Requirement already satisfied: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest->texsoup) (0.1.7)
    Requirement already satisfied: requests>=1.0.0 in /opt/conda/lib/python3.6/site-packages (from coveralls==1.1->texsoup) (2.20.0)
    Requirement already satisfied: docopt>=0.6.1 in /opt/conda/lib/python3.6/site-packages (from coveralls==1.1->texsoup) (0.6.2)
    Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->texsoup) (2.2.2)
    Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->texsoup) (1.11.0)
    Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.12; python_version < "3.8"->pytest->texsoup) (0.6.0)
    Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests>=1.0.0->coveralls==1.1->texsoup) (2018.11.29)
    Requirement already satisfied: idna<2.8,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=1.0.0->coveralls==1.1->texsoup) (2.7)
    Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests>=1.0.0->coveralls==1.1->texsoup) (1.23)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=1.0.0->coveralls==1.1->texsoup) (3.0.4)

```python
import re
import time
import glob
import matplotlib.pyplot as plt
```

# load data

```python
list_of_files = glob.glob('hep-th/2003/*')
```

```python
len(list_of_files)
```

    1019

# find relevant tags in latex

```python
start_time=time.time()
reslts={}
number_of_eq=0
for this_file in list_of_files:
    with open(this_file,'rb') as f:
        data = f.read()

    resp = re.findall('\\\\begin{(?:eqnarray|equation|multiline)}.*?end{(?:eqnarray|equation|multiline)}',str(data),re.DOTALL)
#    print(this_file)
    for eq in resp:
        number_of_eq+=1
        try:
            reslts[this_file].append(eq)
        except KeyError:
            reslts[this_file]=[]
            reslts[this_file].append(eq)
#        print('  ',eq,'\n')

print(round(time.time()-start_time,2),'seconds')
```

    7.63 seconds

```python
number_of_eq
```

    29481

```python
len(reslts.keys())
```

    929

```python
reslts[list(reslts.keys())[0]]
```

    ['\\begin{equation}}\\n\\\\newcommand{\\\\eeq}{\\\\end{equation}',
     '\\begin{eqnarray}}\\n\\\\newcommand{\\\\eeqa}{\\\\end{eqnarray}']

```python
reslts[list(reslts.keys())[1]][0]
```

    '\\begin{equation}\\\\label{planewave}\\n\\\\begin{split}\\nds^2 & = 2dx^+dx^--\\\\m^2\\\\vec{x}^2\\\\bigl(dx^+\\\\bigr)^2+d\\\\vec{x}^2\\\\,,\\\\\\\\\\nF_5 & = 4\\\\m dx^+\\\\wedge\\\\bigl(dx^1\\\\wedge dx^2\\\\wedge dx^3\\\\wedge dx^4+dx^5\\\\wedge dx^6\\\\wedge dx^7\\\\wedge dx^8\\\\bigr)\\\\,.\\n\\\\end{split}\\n\\\\end{equation}'

# visualize results

```python
number_of_eq_per_file=[]
for k,v in reslts.items():
    number_of_eq_per_file.append(len(v))
```

```python
_=plt.hist(number_of_eq_per_file,bins=100)
_=plt.ylabel('number of expressions')
_=plt.xlabel('number of documents')
```

```python
_=plt.hist(number_of_eq_per_file,bins=100)
_=plt.yscale('log', nonposy='clip')
_=plt.ylabel('number of expressions')
_=plt.xlabel('number of documents')
```

# parse the Latex

options:
* http://plastex.sourceforge.net/plastex/sect0025.html
* tex2py
* texsoup

```python
# https://github.com/alvinwan/tex2py
from tex2py import tex2py
```

```python
list_of_files[1]
```

    'hep-th/2003/0304232'

```python
with open(list_of_files[1]) as f: data = f.read()
toc = tex2py(data)
```
toc.valid_tags

```python
from TexSoup import TexSoup
soup = TexSoup(data)
```

```python
soup.equation
```

    \begin{equation}\label{planewave}
    \begin{split}
    ds^2 & = 2dx^+dx^--\m^2\vec{x}^2\bigl(dx^+\bigr)^2+d\vec{x}^2\,,\\
    F_5 & = 4\m dx^+\wedge\bigl(dx^1\wedge dx^2\wedge dx^3\wedge dx^4+dx^5\wedge dx^6\wedge dx^7\wedge dx^8\bigr)\,.
    \end{split}
    \end{equation}

```python
lst = list(soup.find_all('equation'))
```

```python
len(lst) # how many equations are in the document?
```

    79

```python
lst[0] # show the first match
```

    \begin{equation}\label{planewave}
    \begin{split}
    ds^2 & = 2dx^+dx^--\m^2\vec{x}^2\bigl(dx^+\bigr)^2+d\vec{x}^2\,,\\
    F_5 & = 4\m dx^+\wedge\bigl(dx^1\wedge dx^2\wedge dx^3\wedge dx^4+dx^5\wedge dx^6\wedge dx^7\wedge dx^8\bigr)\,.
    \end{split}
    \end{equation}

```python
lst[0][0] # what is the string inside the "begin{equation}"?
```

    '\\label{planewave}\n\\begin{split}\nds^2 & = 2dx^+dx^--\\m^2\\vec{x}^2\\bigl(dx^+\\bigr)^2+d\\vec{x}^2\\,,\\\\\nF_5 & = 4\\m dx^+\\wedge\\bigl(dx^1\\wedge dx^2\\wedge dx^3\\wedge dx^4+dx^5\\wedge dx^6\\wedge dx^7\\wedge dx^8\\bigr)\\,.\n'

```python
lst[1]
```

    \begin{equation}
    \o_n=\sqrt{n^2+\bigl(\m\a'p^+\bigr)^2}\,,\qquad n\in\Nop\,,
    \end{equation}

```python
lst[1][0]
```

    "\n\\o_n=\\sqrt{n^2+\\bigl(\\m\\a'p^+\\bigr)^2}\\,,\\qquad n\\in\\Nop\\,,\n"

```python
for this_eq in soup.find_all('equation'):
    print(this_eq[0])
```

    \label{planewave}
    \begin{split}
    ds^2 & = 2dx^+dx^--\m^2\vec{x}^2\bigl(dx^+\bigr)^2+d\vec{x}^2\,,\\
    F_5 & = 4\m dx^+\wedge\bigl(dx^1\wedge dx^2\wedge dx^3\wedge dx^4+dx^5\wedge dx^6\wedge dx^7\wedge dx^8\bigr)\,.

    \o_n=\sqrt{n^2+\bigl(\m\a'p^+\bigr)^2}\,,\qquad n\in\Nop\,,

    \label{dict}
    \frac{1}{\m}H = \D-J\,,\qquad \frac{1}{\bigl(\m\a'p^+\bigr)^2} = \frac{g^2_{\text{YM}}N}{J^2}\equiv \l'\,,\qquad
    4\pi g_{\text{s}}\bigl(\m\a'p^+\bigr)^2 = \frac{J^2}{N}\equiv g_2

    N\to\infty\quad\text{and}\quad J\to\infty\quad\text{with}\quad\frac{J^2}{N}\quad
    \text{fixed}\,,\quad g_{\text{YM}}\quad\text{fixed}\,,

    S_{\mbox{\scriptsize bos.}(r)}=\frac{e(\a_r)}{4\pi\a'}\int\,d\t\int_0^{2\pi|\a_r|}
    \,d\s_r\bigl[\dot{x}_r^2-x^{\prime\,2}_r-\m^2x_r^2\bigr]\,,

    \begin{split}
    x_r^I(\s_r)& = x_{0(r)}^I+\sqrt{2}\sum_{n=1}^{\infty}
    \bigl(x_{n(r)}^I\cos\frac{n\s_r}{|\a_r|}+x_{-n(r)}^I\sin\frac{n\s_r}{|\a_r|}\bigr)\,,\\
    p_r^I(\s_r) & =\frac{1}{2\pi|\a_r|}\bigl[p_{0(r)}^I+\sqrt{2}\sum_{n=1}^{\infty}
    \bigl(p_{n(r)}^I\cos\frac{n\s_r}{|\a_r|}+p_{-n(r)}^I\sin\frac{n\s_r}{|\a_r|}\bigr)\bigr]\,.

    \label{xp}
    x_{n(r)}^I=i\sqrt{\frac{\a'}{2\o_{n(r)}}}\bigl(a_{n(r)}^I-a_{n(r)}^{I\,\dag}\bigr)\,,\qquad
    p_{n(r)}^I=\sqrt{\frac{\o_{n(r)}}{2\a'}}\bigl(a_{n(r)}^I+a_{n(r)}^{I\,\dag}\bigr)\,.

    =i\d^{IJ}\d_{rs}\d(\s_r-\s_s)\qquad\lr\qquad [a_{n(r)}^I,a_{m(s)}^{J\,\dag}]=\d^{IJ}\d_{nm}\d_{rs}\,.

    S_{\mbox{\scriptsize ferm.}(r)}=\frac{1}{8\pi}\int\,d\t\int_0^{2\pi|\a_r|}\,d\s_r
    [i(\bar{\vt}_r\dot{\vt}_r+\vt_r\dot{\bar{\vt}}_r)
    -\vt_r\vt'_r+\bar{\vt}_r\bar{\vt}'_r-2\m\bar{\vt}_r\Pi\vt_r]\,,

    \begin{split}
    \vt^a_r(\s_r) & =\vt^a_{0(r)}+\sqrt{2}\sum_{n=1}^{\infty}
    \bigl(\vt^a_{n(r)}\cos\frac{n\s_r}{|\a_r|}+\vt^a_{-n(r)}\sin\frac{n\s_r}{|\a_r|}\bigr)\,,\\
    \l^a_r(\s_r) & =\frac{1}{2\pi|\a_r|}\bigl[\l^a_{0(r)}+\sqrt{2}\sum_{n=1}^{\infty}
    \bigl(\l^a_{n(r)}\cos\frac{n\s_r}{|\a_r|}+\l^a_{-n(r)}\sin\frac{n\s_r}{|\a_r|}\bigr)\bigr]\,.

    \{\vt^a_r(\s_r),\l^b_s(\s_s)\}=\d^{ab}\d_{rs}\d(\s_r-\s_s)\qquad\lr\qquad\{\vt^a_{n(r)},\l^b_{m(s)}\}=\d^{ab}\d_{nm}\d_{rs}\,.

    \vt_{n(r)}=\frac{c_{n(r)}}{\sqrt{|\a_r|}}\left[(1+\r_{n(r)}\Pi)b_{n(r)}
    +e(\a_rn)(1-\r_{n(r)}\Pi)b_{-n(r)}^{\dag}\right]\,,\qquad n\in\Zop\,,

    \r_{n(r)}=\r_{-n(r)}=\frac{\o_{n(r)}-|n|}{\m\a_r}\,,\qquad
    c_{n(r)}=c_{-n(r)}=\frac{1}{\sqrt{1+\r_{n(r)}^2}}\,.

    \{b^a_{n(r)},b^{b\,\dag}_{m(s)}\}=\d^{ab}\d_{nm}\d_{rs}\,.

    \Pi =
    \begin{pmatrix}
    \d_{\a_1}^{\b_1}\d_{\a_2}^{\b_2} & 0 \\ 0 & -\d^{\da_1}_{\db_1}\d^{\da_2}_{\db_2}

    \{b_{n(r)\,\a_1\a_2},b^{\b_1\b_2\,\dag}_{m(s)}\}=\d_{\a_1}^{\b_1}\d_{\a_2}^{\b_2}\d_{nm}\d_{rs}\,,\qquad
    \{b_{n(r)\,\da_1\da_2},b^{\db_1\db_2\,\dag}_{m(s)}\}=\d_{\da_1}^{\db_1}\d_{\da_2}^{\db_2}\d_{nm}\d_{rs}\,.

    H_{2(r)}=\frac{1}{\a_r}\sum_{n\in\Zop}\o_{n(r)}\bigl(a_{n(r)}^{I\,\dag}a_{n(r)}^I
    +b^{\a_1\a_2\,\dag}_{n(r)}b_{n(r)\,\a_1\a_2}+b^{\da_1\da_2\,\dag}_{n(r)}b_{n(r)\,\da_1\da_2}\bigr)\,.

    a_{n(r)}|v\ra_r=0\,,\qquad b_{n(r)}|v\ra_r=0\,,\quad n\in\Nop\,.

    \label{comm}
    \begin{split}
    [H,P^I] &= -i\m^2J^{+I}\,,\qquad [H,Q^+]=-\m\Pi Q^+\,,\\
    \{Q^-_{\da},\bar{Q}^-_{\db}\} &=
    2\d_{\dot{a}\dot{b}}H-i\m\bigl(\g_{ij}\Pi\bigr)_{\dot{a}\dot{b}}J^{ij}+i\m\bigl(\g_{i'j'}\Pi\bigr)_{\dot{a}\dot{b}}J^{i'j'}\,.

    \label{q-mode}
    \begin{split}
    Q^-_{(r)} & =\frac{e(\a_r)}{\sqrt{|\a_r|}}\g
    \Bigl(\sqrt{\m}\left[a_{0(r)}(1+e(\a_r)\Pi)+a_{0(r)}^{\dag}(1-e(\a_r)\Pi)\right]\l_{0(r)}\\
    &+\sum_{n\neq 0}\sqrt{|n|}\left[a_{n(r)}P_{n(r)}^{-1}b_{n(r)}^{\dag}
    +e(\a_r)e(n)a_{n(r)}^{\dag}P_{n(r)}b_{-n(r)}\right]\Bigr)\,,

    P_{n(r)}\equiv\frac{1-\r_{n(r)}\Pi}{\sqrt{1-\r_{n(r)}^2}}
    =\frac{1+\Pi}{2}U_{|n|(r)}^{1/2}+\frac{1-\Pi}{2}U_{|n|(r)}^{-1/2}\,,\qquad
    U_{n(r)}\equiv\frac{\o_{n(r)}-\m\a_r}{n}\,.

    \label{hexp}
    H=H_2+g_{\text{s}}H_3+\cdots\,,

     = -i\m^2J^{+I}\quad\Longrightarrow\quad[H_3,P^I]=0\,,

    \label{q+}
    [H,Q^+]=-\m\Pi Q^+\quad\Longrightarrow\quad[H_3,Q^+]=0\,,

    |V\ra\equiv|E_a\ra|E_b\ra\d\bigl(\sum_{r=1}^3\a_r\bigr)\,,

    \label{bv}
    |E_a\ra = \exp\left(\frac{1}{2}
    \sum_{r,s=1}^3\sum_{m,n\in\Zop}a^{I\,\dag}_{m(r)}\bar{N}^{rs}_{mn}a^{I\,\dag}_{n(s)}
    \right)|v\ra_{123}\,,

    \label{eb2}
    |E_b\ra=\exp\left(
    \sum_{r,s=1}^3\sum_{m,n\ge0}
    \bigl(b^{\a_1\a_2\,\dag}_{-m(r)}b^{\dag}_{n(s)\,\a_1\a_2}+b^{\da_1\da_2\,\dag}_{m(r)}b^{\dag}_{-n(s)\,\da_1\da_2}\bigr)
    \bar{Q}^{rs}_{mn}\right)|v\ra_{123}\,,

    \sqrt{2}\eta\,Q\equiv Q^-+i\bar{Q}^-\,,\qquad\text{and}\qquad
    \sqrt{2}\bar{\eta}\,\wt{Q}=Q^--i\bar{Q}^-

    \{Q_{\dot{a}},\wt{Q}_{\dot{b}}\}=0\,,\quad
    \{Q_{\dot{a}},Q_{\dot{b}}\}=\{\wt{Q}_{\dot{a}},\wt{Q}_{\dot{b}}\}
    =2\d_{\dot{a}\dot{b}}H-i\m\bigl(\g_{ij}\Pi\bigr)_{\dot{a}\dot{b}}J^{ij}+i\m\bigl(\g_{i'j'}\Pi\bigr)_{\dot{a}\dot{b}}J^{i'j'}\,.

    K^I \equiv \sum_{r=1}^3\sum_{n\in\Zop}K_{n(r)}a_{n(r)}^{I\,\dag}\,,\qquad
    \K^I \equiv \sum_{r=1}^3\sum_{n\in\Zop}\K_{n(r)}a_{n(r)}^{I\,\dag}\,,

    K_{n(r)} \equiv (1-4\m\a K)^{1/2}\begin{cases} F_{0(r)}\,, & n=0 \\
    F_{n(r)}\,, & n>0 \\
    iU_{n(r)}F_{n(r)}\,, & n<0

    \label{Y}
    Y^{\a_1\a_2} = \sum_{r=1}^3\sum_{n\ge 0}\bar{G}_{n(r)}b_{n(r)}^{\dag\,\a_1\a_2}\,,\qquad
    Z^{\da_1\da_2} = \sum_{r=1}^3\sum_{n\ge 0}\bar{G}_{n(r)}b_{-n(r)}^{\dag\,\da_1\da_2}\,,

    \label{G}
    \bar{G}_{(r)} =
    \sqrt{-\frac{\a'}{\a}}(1-4\m\a K)^{1/2}\sqrt{|\a_r|}U_{(r)}^{1/2}C^{-1/2}F_{(r)}\,,

    \label{qy}
    \begin{split}
    \eta\sqrt{\frac{-2\a}{\a'}}\sum_{r=1}^3\{Q_{(r)},Y+iZ\} & = i\g^IK^I\,,\\
    \bar{\eta}\sqrt{\frac{-2\a}{\a'}}\sum_{r=1}^3\{\wt{Q}_{(r)},Y-iZ\} & = -i\g^I\wt{K}^I\,.

    ^I\equiv\sum_{r=1}^3\sum_{n\in\Zop}K^*_{n(r)}a_{n(r)}^{I\,\dag} = \K^I,\quad
    Y^*\equiv\sum_{r=1}^3\sum_{n\ge 0}\bar{G}^*_{n(r)}b_{n(r)}^{\dag}=Y,\quad
    Z^*\equiv\sum_{r=1}^3\sum_{n\ge 0}\bar{G}^*_{n(r)}b_{-n(r)}^{\dag} = Z,

    \label{ansatz1}
    |Q_{3\,\b_1\db_2}\ra =
    \bigl(f^i_{\b_1\db_2}(Y,Z)\K^i+g^{i'}_{\b_1\db_2}(Y,Z)\K^{i'}\bigr)|V\ra\,,

    \label{y2}
    Y^2_{\a_1\b_1} \equiv Y_{\a_1\a_2}Y^{\a_2}_{\b_1}\,,\qquad Y^2_{\a_2\b_2} \equiv Y_{\a_1\a_2}Y^{\a_1}_{\b_2}\,,

    \label{y3}
    Y^3_{\a_1\b_2} \equiv Y^2_{\a_1\b_1}Y^{\b_1}_{\b_2}=-Y^2_{\b_2\a_2}Y^{\a_2}_{\a_1}\,,

    Y^4_{\a_1\b_1} \equiv Y^2_{\a_1\g_1}{Y^2}^{\g_1}_{\b_1}=-\frac{1}{2}\e_{\a_1\b_1}Y^4\,,\qquad
    Y^4_{\a_2\b_2} \equiv Y^2_{\a_2\g_2}{Y^2}^{\g_2}_{\b_2}=\frac{1}{2}\e_{\a_2\b_2}Y^4\,,

    \label{y4}
    Y^4 \equiv Y^2_{\a_1\b_1}{Y^2}^{\a_1\b_1}=-Y^2_{\a_2\b_2}{Y^2}^{\a_2\b_2}\,.

    \bar{\eta}\sqrt{\frac{\a'}{-2\a}}|Q_{3\,\b_1\db_2}\ra =
    \Bigl(s_{\dg_1\db_2}(Z)t_{\b_1\g_1}(Y)\K^{\dg_1\g_1}+is_{\b_1\g_2}(Y)t^*_{\db_2\dg_2}(Z)\K^{\dg_2\g_2}\Bigr)|V\ra\,,

    -\eta\sqrt{\frac{\a'}{-2\a}}|Q_{3\,\db_1\b_2}\ra =
    \Bigl(s^*_{\g_1\b_2}(Y)t^*_{\db_1\dg_1}(Z)\K^{\dg_1\g_1}+is^*_{\db_1\dg_2}(Z)t_{\b_2\g_2}(Y)\K^{\dg_2\g_2}\Bigr)|V\ra\,.

    \K^{\dg_1\g_1} \equiv \K^i{\s^i}^{\dg_1\g_1}\,,\qquad
    \K^{\dg_2\g_2} \equiv \K^{i'}{\s^{i'}}^{\dg_2\g_2}\,,

    s(Y) \equiv Y+\frac{i}{3}Y^3\,,\qquad t(Y) \equiv \e+iY^2-\frac{1}{6}Y^4\,.

    ^{ij} \equiv \s^{ij}_{\a_1\b_1}{Y^2}^{\a_1\b_1}\,,\quad
    {Z^2}^{ij} \equiv \s^{ij}_{\da_1\db_1}{Z^2}^{\da_1\db_1}\,,\quad
    \bigl(Y^2Z^2\bigr)^{ij} \equiv {Y^2}^{k(i}{Z^2}^{j)k}

    \label{nij}
    |n\ra\equiv\a^{I\,\dag}_{n(3)}\a^{J\,\dag}_{-n(3)}|v\ra_3\,.

    \label{de1}
    \d E_n^{(2)}\la n|n\ra = g_2^2\sum_{1,2}
    \frac{1}{2}\frac{\left|\la n|H_3|1,2\ra\right|^2}{E_n^{(0)}-E_{1,2}^{(0)}}
    -\frac{g_2^2}{4}\la n|Q_{3\,\b_1\db_1}Q_3^{\b_1\db_1}|n\ra\,.

    \label{tr}
    x^+ \to -x^+\,,\qquad x^- \to - x^-\,,\qquad \m \to -\m\,,

    \m \to -\m\,,\qquad \Pi \to -\Pi

    \label{de2}
    {\mc N}\d E_n^{(2)}=-g_2^2\int_{-1}^0\frac{d\b}{\b(\b+1)}
    \sum_{\text{modes}}\left[
    \frac{1}{2}\frac{\left|\la n|H_3|1,2\ra\right|^2}{E_n^{(0)}-E_{1,2}^{(0)}}
    -\frac{1}{4}\la n|Q_{3\,\b_1\db_1}|1,2\ra\la 1,2|Q_3^{\b_1\db_1}|n\ra\right]\,.

    \begin{split}
    \la n|H_3|\a_{0(r)}^{\dag\,k}\a_{0(s)}^{\dag\,l}|v\ra_{12}& \sim
    \m\l'\frac{\sin^2 n\pi\b}{2\pi^2}\left(\d^{rs}+\frac{\sqrt{\a_r\a_s}}{\a_3}\right)S^{ijkl}\,,\\
    \la n|H_3|\a_{k(r)}^{\dag\,k}\a_{-k(r)}^{\dag\,l}|v\ra_{12} & \sim
    \m\l'\b(\b+1)\frac{\a_3}{\a_r}\frac{\sin^2 n\pi\b}{2\pi^2}S^{ijkl}\,,

    S^{ijkl}\equiv T^{ijkl}+\frac{1}{4}\d^{ij}T^{kl}\,,\quad
    T^{ijkl}=\d^{ik}\d^{jl}+\d^{jk}\d^{il}-\frac{1}{2}\d^{ij}\d^{kl}\,,\quad
    T^{kl} = -2\d^{kl}

    \frac{\m g_2^2\l'}{4\pi^2}\frac{15}{64\pi^2n^2}\sum_{k,l}S^{ijkl}S^{ijkl}=
    \frac{\m g_2^2\l'}{4\pi^2}\frac{15}{16\pi^2n^2}\Bigl[\frac{1}{2}\bigl(1+\frac{1}{2}\d^{ij}\bigr)+\frac{1}{4}\d^{ij}\Bigr]\,.

    \label{9-1}
    \frac{\m g_2^2\l'}{4\pi^2}\frac{15}{16\pi^2n^2}\,.

    -\pi\a'\m^2\la n|K^M\K^N|E_a\ra
    \sim\frac{\m\sqrt{\l'}}{2\pi}\b(\b+1)\sin^2n\pi\b\left(\d^{MI}\d^{NJ}+\d^{NI}\d^{MJ}\right)\,.

    -{}_{123}\la v|b_{k(r)\,\a_1\a_2}b_{-k(s)\,\da_1\da_2}Y^{\g_1\g_2}Z^{\dg_1\dg_2}|E_b\ra
    =\d_{\a_1}^{\g_1}\d_{\a_2}^{\g_2}\d_{\da_1}^{\dg_1}\d_{\da_2}^{\dg_2}
    \bar{G}_{k(r)}\bar{G}_{k(s)}\,.

    \frac{\m g_2^2\l'}{4\pi^2}\frac{3}{8\pi^2 n^2}
    -\frac{\m g_2^2\l'}{4n\pi^3}\int_{-1}^0 d\b\,\sin^4n\pi\b\cot n\pi\b(\b+1)
    =\frac{\m g_2^2\l'}{4\pi^2}\frac{15}{16\pi^2 n^2}\frac{1}{2}\,,

    \eta|Q_{3\,\b_1\db_2}\ra \sim
    -\sqrt{-\frac{\a'}{8\a_3^3}}\sqrt{-\b(\b+1)}\bigl(Y_{\b_1\g_2}\K^{\g_2}_{\db_1}-iZ_{\dg_1\b_2}\K^{\dg_1}_{\b_1}\bigr)|E_a\ra|E_b\ra\,,

    \frac{1}{2}\frac{\m g_2^2\l'}{4\pi^2}\left(\frac{1}{12}+\frac{35}{32n^2\pi^2}\right)\,,

    \frac{\m g_2^2\l'}{4\pi^2}\left(\frac{1}{12}+\frac{5}{32n^2\pi^2}\right)
    \Bigl[\frac{1}{2}\bigl(1+\frac{1}{2}\d^{ij}\bigr)+\frac{1}{4}\d^{ij}\Bigr]\,,

    \d E_n^{(2)}=\frac{\m g_2^2\l'}{4\pi^2}\left(\frac{1}{12}+\frac{35}{32n^2\pi^2}\right)\,.

    _v \longrightarrow ({\bf 4},{\bf 1})\oplus ({\bf 1},{\bf 4})\,,\qquad
    {\bf 8}_s \longrightarrow ({\bf 2},{\bf 2}) \oplus ({\bf 2'},{\bf 2'})\,,\qquad
    {\bf 8}_c \longrightarrow ({\bf 2},{\bf 2'}) \oplus ({\bf 2'},{\bf 2})\,,

    \s^i_{\a\da}=\bigl(i\t^1,i\t^2,i\t^3,-1\bigr)_{\a\da}

    \s^i_{\a\da} = \e_{\a\b}\e_{\da\db}{\s^i}^{\db\b}
    \equiv \e_{\a\b}{\s^i}^{\b}_{\da} \equiv \e_{\da\db}{\s^i}^{\db}_{\a}\,.

    \s^i_{\a\da}{\s^j}^{\da\b}+\s^j_{\a\da}{\s^i}^{\da\b}
    =2\d^{ij}\d_{\a}^{\b}\,,\qquad
    {\s^i}^{\da\a}\s^j_{\a\db}+{\s^j}^{\da\a}\s^i_{\a\db}=
    2\d^{ij}\d^{\da}_{\db}\,.

    \Pi_{ab} =
    \begin{pmatrix} \bigl(\s^1\s^2\s^3\s^4\bigr)_{\a_1}^{\b_1}\d_{\a_2}^{\b_2} & 0 \\
    0 & \bigl(\s^1\s^2\s^3\s^4\bigr)^{\da_1}_{\db_1}\d^{\da_2}_{\db_2}

    \label{1Y234}
    \begin{split}
    \sum_r[Q_{(r)\,\a_1\da_2},Y^2_{\b_1\g_1}] & =
    -i\bigl(\e_{\a_1\b_1}Y_{\g_1\g_2}+\e_{\a_1\g_1}Y_{\b_1\g_2}\bigr)K^{\g_2}_{\da_2}\,,\\
    \sum_r[Q_{(r)\,\a_1\da_2},Y^2_{\b_2\g_2}] & =
    i\bigl(Y_{\a_1\b_2}K_{\g_2\da_2}+Y_{\a_1\g_2}K_{\b_2\da_2}\bigr)\,,\\
    \sum_r\{Q_{(r)\,\a_1\da_2},Y^3_{\b_1\g_2}\} & =
    3iY_{\a_1\g_2}Y_{\b_1\d_2}K^{\d_2}_{\da_2}\,,\\
    \sum_r[Q_{(r)\,\a_1\da_2},Y^4] & =
    4iY^3_{\a_1\a_2}K^{\a_2}_{\da_2}\,,

    \label{2Y234}
    \begin{split}
    \sum_r[Q_{(r)\,\db_1\b_2},Y^2_{\a_1\g_1}] & =
    -i\bigl(Y_{\g_1\b_2}K_{\a_1\db_1}+Y_{\a_1\b_2}K_{\g_1\db_1}\bigr)\,,\\
    \sum_r[Q_{(r)\,\db_1\b_2},Y^2_{\a_2\g_2}] & =
    -i\bigl(\e_{\a_2\b_2}Y_{\g_1\g_2}+\e_{\g_2\b_2}Y_{\g_1\a_2}\bigr)K^{\g_1}_{\db_1}\,,\\
    \sum_r\{Q_{(r)\,\db_1\b_2},Y^3_{\a_1\g_2}\} & =
    3iY_{\a_1\b_2}Y_{\g_1\g_2}K^{\g_1}_{\db_1}\,,\\
    \sum_r[Q_{(r)\,\db_1\b_2},Y^4] & =
    -4iY^3_{\a_1\b_2}K^{\a_1}_{\db_1}\,,

    \label{1Z234}
    \begin{split}
    \sum_r[Q_{(r)\,\a_1\da_2},Z^2_{\db_1\dg_1}] & =
    K_{\a_1\db_1}Z_{\dg_1\da_2}+K_{\a_1\dg_1}Z_{\db_1\da_2}\,,\\
    \sum_r[Q_{(r)\,\a_1\da_2},Z^2_{\db_2\dg_2}] & =
    -\bigl(\e_{\da_2\db_2}Z_{\dg_1\dg_2}+\e_{\da_2\dg_2}Z_{\dg_1\db_2}\bigr)K_{\a_1}^{\dg_1}\,,\\
    \sum_r\{Q_{(r)\,\a_1\da_2},Z^3_{\db_1\dg_2}\} & =
    -3Z_{\db_1\da_2}Z_{\dg_1\dg_2}K^{\dg_1}_{\a_1}\,,\\
    \sum_r[Q_{(r)\,\a_1\da_2},Z^4] & =
    4Z^3_{\da_1\da_2}K^{\da_1}_{\a_1}\,,

    \label{2Z234}
    \begin{split}
    \sum_r[Q_{(r)\,\db_1\b_2},Z^2_{\da_1\dg_1}] & =
    -\bigl(\e_{\db_1\da_1}Z_{\dg_1\dg_2}+\e_{\db_1\dg_1}Z_{\da_1\dg_2}\bigr)K_{\b_2}^{\dg_2}\,,\\
    \sum_r[Q_{(r)\,\db_1\b_2},Z^2_{\da_2\dg_2}] & =
    K_{\b_2\da_2}Z_{\db_1\dg_2}+K_{\b_2\dg_2}Z_{\db_1\da_2}\,,\\
    \sum_r\{Q_{(r)\,\db_1\b_2},Z^3_{\da_1\dg_2}\} & =
    3Z_{\db_1\dg_2}Z_{\da_1\dd_2}K^{\dd_2}_{\b_2}\,,\\
    \sum_r[Q_{(r)\,\db_1\b_2},Z^4] & =
    4Z^3_{\db_1\dg_2}K^{\dg_2}_{\b_2}\,.

    -4\frac{\m\a}{\a'}\e_{\a_1\b_1}\e_{\da_2\db_2}
    \Bigl(a_{3,0}Y^4+ia_{0,3}Z^4+(a_{3,4}+ia_{4,3})Y^4Z^4\Bigr)\,,\quad
    \text{and}\quad a_{2,1}=ia_{1,2}\,.

    2Y_{[\a_1\g_2}K^{\dg_1}_{\b_1]} = \e_{\a_1\b_1}Y^{\g_1}_{\g_2}K^{\dg_1}_{\g_1}\,,

    \label{trick}
    \begin{pmatrix} b_{n(r)} \\ b_{-n(r)}

    \begin{split}
    \sum_r[Q_{(r)\,\a_1\da_2},K_{\b_1\db_1}] & =
    -4\frac{\a}{\a'}\e_{\b_1\a_1}\bigl(\p_{\t}+\p_{\s}\bigr)Z_{\db_1\da_2}\,,\\
    \sum_r[Q_{(r)\,\da_1\a_2},K_{\b_1\db_1}] & =
    -4i\frac{\a}{\a'}\e_{\db_1\da_1}\bigl(\p_{\t}+\p_{\s}\bigr)Y_{\b_1\a_2}\,,

    \begin{split}
    \sum_r[Q_{(r)\,\a_1\da_2},K_{\b_2\db_2}] & =
    -4i\frac{\a}{\a'}\e_{\da_2\db_2}\bigl(\p_{\t}+\p_{\s}\bigr)Y_{\a_1\b_2}\,,\\
    \sum_r[Q_{(r)\,\da_1\a_2},K_{\b_2\db_2}] & =
    4\frac{\a}{\a'}\e_{\a_2\b_2}\bigl(\p_{\t}+\p_{\s}\bigr)Z_{\da_1\db_2}\,,

    \label{dV}
    \p_{\s}|V\ra = -\frac{i}{4}
    \left[\frac{\a'}{\a}\bigl(K^IK^I-\K^I\K^I\bigr)-
    4\bigl(Y_{\a_1\a_2}\p_{\t}Y^{\a_1\a_2}-Z_{\da_1\da_2}\p_{\t}Z^{\da_1\da_2}\bigr)\right]|V\ra\,.

    \lim_{\s\to\pi\a_1}Y(\s)|V\ra = Y|V\ra\,,\qquad \lim_{\s\to\pi\a_1}Z(\s)|V\ra = Z|V\ra\,.

    \lim_{\s\to\pi\a_1}Y(\s)|V\ra = f(\m)(1-4\m\a K)^{-1/2}Y|V\ra\,,

    f(\m)\equiv-2\frac{\sqrt{-\a}}{\a_1}\lim_{e\to0}\e^{1/2}
    \sum_{n=1}^{\infty}(-1)^nn\cos(n\e/\a_1)\bar{N}^1_n=(1-4\m\a K)^{1/2}.

```python

```


```python
import theme
theme.load_style()
```

```python
import sympy as sp
sp.init_printing()
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

# Lesson 16: The Method of Manufactured Solutions

This lecture by Tim Fuller is licensed under the
Creative Commons Attribution 4.0 International License.  All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

# Validation and Verification

## Validation Question
How do we know if the equations we have been solving are the right equations?

* Compare with data or with more sophisticated models.

## Verification Question
How do we know that we are solving the equations right?

* Compare the approximate solution with an exact solution whenever possible.  Otherwise, perform a variety of "sanity checks".  One such sanity check is the method of manufactured solutions.

# Method of Manufactured Solutions (MMS)

The method of manufactured solutions is a straightforward, yet powerful, method to verify that the finite element equations are being solved correctly.  In the MMS, we start with a "manufactured solution" $u$ and determine the form of the sourcing functions required to produce the solution by applying the governing equations to it.  The sourcing functions are then applied to the code and solution checked.

For example, consider the following boundary value problem

$$
\frac{d\sigma}{dx} + b = 0
$$
$$u(0) = 0$$
$$\sigma(L)=0$$

Let $\sigma = E\epsilon$ (linear elasticity) and suppose $u = f(x)$.  Applying the governing equations to $u$ we get

$$
\frac{d\sigma}{dx} + b = E\frac{d^2f}{dx^2} + b = 0
\quad \longrightarrow \quad b = -E\frac{d^2f}{dx^2}
$$

Enforcing boundary conditions supplies additional constraints on $f$:

$$f(0) = 0$$
$$E\frac{df}{dx}=0$$

Other constraints (as needed) can be imposed on $f$.

## Example

Let $u = u_0 + u_1 x + u_2 x^2 + u_2 x^3$.  The boundary conditions require that

$$u(0) = u_0 = 0$$
$$\sigma(L) = E\frac{du}{dx} =
E\frac{d}{dx}\left(u_0 + u_1 x + u_2 x^2 + u_2 x^3\right) = 0$$

We have two equations, but three unknowns.  We choose the third equation by arbitrarily scaling the displacement at L to 1

$$u(L) = u_0 + u_1 L + u_2 L^2 + u_3 L^3 = 1$$

We solve the preceding equations below using sympy

```python
# Manufactured solution
uL = 1
x, L, b, s, E = sp.symbols("x L b sigma E")
u0, u1, u2 = sp.symbols("u0 u1 u2")
a = u0 + u1 * x ** 2 + u2 * x ** 3
eps = sp.diff(a, x)
sig = E * eps * (eps + 2.) / (2. * eps + 2.)
coeffs = sp.solve([a.subs(x, 0), sig.subs(x, L), a.subs(x, L) - uL],
                  (u0, u1, u2))
coeffs = dict(zip((u0, u1, u2), coeffs[0]))
ua = a.subs(coeffs)
eps = ua.diff(x)
sig = E * eps
b = -sig.diff(x)
b
```

Yuck!  We now use the $b$ previously determined as an input to the simple linear FE program below to verify that we get back the manufactured solution $u$.

```python
ROOT3 = sqrt(3.)
def wundee(xa, xb, num_elem, E, q):
    """Solves

      ds                            du
      -- + q = 0, where s = Ee, e = --
      dx                            dx

      subject to

      u(0) = 0, s(L) = 0

    Parameters
    ----------
    xa, xb : real
        Origin and end of bar
    num_elem : int
        Number of elements
    E : real
        Young's modulus
    q : callable
        Body force per unit area

    Returns
    -------
    X, u : array
        Nodal coordinates and displacements

    """
    # Linear 1D mesh
    n = num_elem + 1
    X = linspace(xa, xb, n)
    connect = array([[i, i+1] for i in range(num_elem)])

    # Global stiffness and force
    K = zeros((n, n))
    F = zeros(n)

    # Shape functions and derivatives, Jacobian
    N = lambda xi: array([1 - xi, 1 + xi]) / 2.
    dN = lambda xi: array([-1., 1.]) / 2.
    Jac = lambda xi, xc: dot(dN(xi), xc)
    x = lambda xi, xc: xc[0] + (xc[1] - xc[0]) / 2. * (1. + xi)

    # assemble global stiffness
    for (e, nodes) in enumerate(connect):
        np = len(nodes)
        xc = X[nodes]
        ke = zeros((2, 2))
        fe = zeros(2)
        for xi in (-1./ROOT3, 1./ROOT3):
            # Convert shape function derivatives to derivatives
            # wrt global coords
            J = Jac(xi, xc)
            dNdx = dN(xi) / J
            ke += E * J * outer(dNdx, dNdx)
            fe += q(x(xi, xc)) * N(xi) * J

        # add contributions to global matrices
        for a in range(np):
            I = nodes[a]
            F[I] += fe[a]
            for b in range(np):
                J = nodes[b]
                K[I, J] += ke[a, b]

    # Apply boundary conditions
    K[0,0] = 1e30
    F[0] = 0.
    u = linalg.solve(K, F)

    return X, u
```

```python
xa, xb, n, E_ = 0., 1., 5, 10
xc = linspace(xa, xb)
plot(xc, sp.lambdify(x, ua.subs({E: E_, L: xb}))(xc),
     lw=2, label='Analytic Solution')

q = sp.lambdify(x, b.subs({E: E_, L: xb}))
xp, u = wundee(xa, xb, n, E_, q)
plot(xp, u, '-.', lw=2, label='FE Solution')
legend(loc='best');
```


```
import requests
from bs4 import BeautifulSoup
from pathlib import Path
import time
import re
from datetime import datetime
import os
from nltk.corpus import stopwords
import pandas as pd
import math
import json
import string
import nltk
from nltk.stem import LancasterStemmer
from nltk.stem import PorterStemmer,WordNetLemmatizer
from nltk.tokenize import word_tokenize
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')
```

    [nltk_data] Downloading package punkt to /root/nltk_data...
    [nltk_data]   Package punkt is already up-to-date!
    [nltk_data] Downloading package stopwords to /root/nltk_data...
    [nltk_data]   Package stopwords is already up-to-date!
    [nltk_data] Downloading package wordnet to /root/nltk_data...
    [nltk_data]   Package wordnet is already up-to-date!

    True

```
from google.colab import drive
drive.mount('/content/drive')
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

```
!unzip /content/drive/MyDrive/ADM/HM3/pages.zip
```

# Functions

```
def handler_date(string_date):
  if string_date.strip() == 'Not available': return ""
  if len(string_date.strip()) == 4:
    return datetime.strptime(string_date.lower().strip(),"%Y")
  elif len(string_date.strip()) == 8:
    return datetime.strptime(string_date.lower().strip(),"%b %Y")
  return datetime.strptime(string_date.lower().strip(),"%b %d %Y")
```

```
# function that returns number of episodes, type anime, releaseDate,endDate
def get_information(soup):
  lista = soup.find_all("div", class_="spaceit_pad")
  type_anime =""
  episode = 0
  releaseDate =""
  endDate = ""
  for el in lista:
    if el.find('span'):
      if el.span.text == "Type:":
        type_anime = el.text.replace("\n","").replace("Type:","").strip()
      elif el.span.text == "Episodes:":
        episode = [int(s) for s in el.text.split() if s.isdigit()]
        if len(episode) > 0:
          episode = episode[0]
        else:
          episode = ""
      elif el.span.text == "Aired:":
        line = re.sub("(\nAired:\n) | *(\n)", '', el.text).split('to')
        if len(line) > 0:
          # convert string to datatime
          releaseDate = handler_date(line[0].replace(",",""))
          if len(line) > 1 and len(line[1].strip()) > 5:
            endDate = handler_date(line[0].replace(",",""))
  return type_anime,episode,releaseDate,endDate
```

```
# generic function that return the text that contain in a specific tag and specific class
def find_element_by_class(class_name,tag_name):
  lista = soup.find_all(tag_name, class_=class_name)
  final_list = []
  set_control = set()
  for el in lista:
      if el.a.text not in set_control:
        set_control.add(el.a.text)
        final_list.append(el.a.text)
  return final_list
```

```
# function that return the Related Anime that contain in a specific tag and specific class
def find_anime(class_name,tag_name,child_class,bool_href):
  lista = soup.find(tag_name, class_=class_name).find_all(child_class,href=bool_href)
  final_list = []
  set_control = set()
  for el in lista:
    if el.text not in set_control:
          set_control.add(el.text)
          final_list.append(el.text)
  return final_list
```

```
# function that return the staff
def find_staff(tag_name, text, tag_sibling, tag_full_name, tag_role):
  staff_table = soup.find(tag_name, text=text).parent.find_next_sibling()
  lista = []
  for el in staff_table.find_all(tag_sibling):
    full_name = el.find_all(tag_full_name)[1].text
    role = el.find(tag_role).text
    lista.append([full_name,role])
  return lista
```

```
# function that elaborate a file html and return a line that contains some information about the page
def descrip_html(path):
  with open(path) as myfile:
      soup = BeautifulSoup(myfile)

  title_name =soup.find("h1", class_="title-name").strong.text if soup.find("h1", class_="title-name") is not None else ""
  type_anime,episode,releaseDate,endDate = get_information(soup)
  score = soup.find("div", class_="fl-l score").text if soup.find("div", class_="fl-l score") is not None else ""
  num_members = soup.find("span", class_="numbers members").strong.text.replace(",","") if soup.find("span", class_="numbers members") is not None else ""
  userCount =  soup.find("span", itemprop="ratingCount").text if soup.find("span", itemprop="ratingCount") is not None else ""
  ranked = soup.find("span", class_="numbers ranked").strong.text.replace("#","") if soup.find("span", class_="numbers ranked") is not None else ""
  popularity = soup.find("span", class_="numbers popularity").strong.text.replace("#","") if soup.find("span", class_="numbers popularity") is not None else ""
  description = soup.find("p", itemprop="description").get_text(strip= True) if soup.find("p", itemprop="description") is not None else ""
  anime = find_anime("anime_detail_related_anime","table","a",True) if find_anime("anime_detail_related_anime","table","a",True) is not None else ""
  actor = find_element_by_class("h3_characters_voice_actors","h3") if find_element_by_class("h3_characters_voice_actors","h3") is not None else ""
  voice = find_element_by_class("va-t ar pl4 pr4","td") if find_element_by_class("va-t ar pl4 pr4","td") is not None else ""
  staff = find_staff("h2", "Staff","tr","a", "small") if find_staff("h2", "Staff","tr","a", "small") is not None else ""
  url = soup.find("meta", property="og:url")["content"]
  return title_name + "\t"+ type_anime + "\t" + str(episode) + "\t" + str(releaseDate) + "\t" + str(endDate) + \
"\t" + str(num_members) + "\t" + str(score) + "\t" + str(userCount) + "\t" + str(ranked) + "\t" \
+ str(popularity) + "\t" + description + "\t" + str(anime) + "\t " + str(actor) + "\t" + str(voice) + "\t" + str(staff) + "\t"+url
```

#REQ 1

## REQ 1.1

```
# get all url from site
list_url = []
for i in range(0,19150,50):
  if i % 1000 == 0:
    print(i)
  data = requests.get('https://myanimelist.net/topanime.php?limit='+str(i))
  soup = BeautifulSoup(data.text)
  lista = soup.find_all("h3", class_="anime_ranking_h3")
  for element in lista:
    list_url.append(element.find('a')['href'])
```

```
# save file
with open('links.txt', 'w') as f:
    for line in list_url:
        f.write(line)
        f.write('\n')
```

## REQ 1.2

```
# function that return all page html in the ranking
#there are different numbers, like page number, because html was downloaded in different sessions, because we were blocked by the site
page_number = 101
directory = "/content/pages/"
with open("/content/links.txt") as myfile:
    directory_page_number = ""
    for i,line in enumerate(myfile):
        if i > 2499:
          if i % 50==0:
            time.sleep(5)
            directory_page_number = directory + str(page_number) + "/"
            Path(directory_page_number).mkdir(parents=True, exist_ok=True)
            page_number += 1
          # if the site blocks us, we sleep and try again after 20 seconds
          while(True):
            data = requests.get(line).text
            soup = BeautifulSoup(data).find_all("div", class_="error")
            if (len(soup) > 0):
              time.sleep(20)
            else:
              break
          with open(directory_page_number + 'article_'+str(i)+'.html', 'w') as f:
            f.write(data)

```

```
!unzip /content/drive/MyDrive/ADM/HM3/pages.zip
```

```
!rm -r -f /content/pages_tsv
```

```
array = !ls -w1 pages_tsv
len(array)
```

    383

## REQ 1.3

```
#we process the html file by extracting the data we need and we save it in a tsv file
dir = "/content/pages"
new_dir = "/content/pages_tsv/"
for page_number, page in enumerate(os.listdir(dir),start=1):
  directory_page_number = new_dir + str(page) + "/"
  Path(directory_page_number).mkdir(parents=True, exist_ok=True)
  if os.path.isdir(dir+"/"+page) and len(os.listdir(dir+"/"+page)):
    html_files = os.listdir(dir+"/"+page)
    for anime in html_files:
      if anime.endswith('.html'):
        final_string =  descrip_html(dir +"/"+ page+ "/" + anime) + "\n"
        with open(directory_page_number + "anime_" + anime[anime.find("_")+1: anime.find(".")] +".tsv","w") as myfile:
          myfile.write(final_string)
```

```
#we read all the directories where the tsv files are contained and merge them into a single tsv file to upload in pandas
with open("pages_tsv1.tsv","w") as file:
  dir = "/content/pages_tsv/"
  for page_number, page in enumerate(os.listdir(dir),start=1):
    if os.path.isdir(dir+"/"+page) and len(os.listdir(dir+"/"+page)):
      html_files = os.listdir(dir+"/"+page)
      for anime in html_files:
        if anime.endswith('.tsv'):
          with open(dir+page+"/"+anime, "r") as tsv_file:
              file.write(tsv_file.read())
```

# Clear data

```
# load data in pandas
col=["animeTitle","animeType","animeNumEpisode","releaseDate","endDate","animeNumMembers",\
           "animeScore","animeUsers","animeRank","animePopularity","animeDescription",\
           "animeRelated","animeCharacters", "animeVoices","animeStaff","url"]
df = pd.read_csv("/content/pages_tsv1.tsv",sep='\t', names=col, parse_dates=col[3:5])
```

```
df.shape
```

    (19128, 16)

```
# As we can, we must to remove all null value
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 19128 entries, 0 to 19127
    Data columns (total 16 columns):
     #   Column            Non-Null Count  Dtype
    ---  ------            --------------  -----
     0   animeTitle        19126 non-null  object
     1   animeType         19126 non-null  object
     2   animeNumEpisode   18619 non-null  float64
     3   releaseDate       18760 non-null  datetime64[ns]
     4   endDate           8551 non-null   datetime64[ns]
     5   animeNumMembers   19126 non-null  float64
     6   animeScore        13438 non-null  float64
     7   animeUsers        13438 non-null  float64
     8   animeRank         17305 non-null  float64
     9   animePopularity   19126 non-null  float64
     10  animeDescription  19120 non-null  object
     11  animeRelated      19128 non-null  object
     12  animeCharacters   19128 non-null  object
     13  animeVoices       19128 non-null  object
     14  animeStaff        19128 non-null  object
     15  url               19128 non-null  object
    dtypes: datetime64[ns](2), float64(6), object(8)
    memory usage: 2.3+ MB

```
# number of null value
df.isnull().sum()
```

    animeTitle              2
    animeType               2
    animeNumEpisode       509
    releaseDate           368
    endDate             10577
    animeNumMembers         2
    animeScore           5690
    animeUsers           5690
    animeRank            1823
    animePopularity         2
    animeDescription        8
    animeRelated            0
    animeCharacters         0
    animeVoices             0
    animeStaff              0
    url                     0
    dtype: int64

```
len(df.animeUsers.unique())
```

    7017

```
df["animeScore"] = df["animeScore"].where(pd.notnull(df["animeScore"]), None)
df["animeUsers"] = df["animeUsers"].where(pd.notnull(df["animeUsers"]), None)
df["animeRank"] = df["animeRank"].where(pd.notnull(df["animeRank"]), None)
```

```
df["animeScore"].isnull().sum()
```

    5690

```
# as we can see, animeTitle, animeNumMembers, animePopularity etc are null, so we need to remove this line
df[df.animeNumMembers.isnull()]
```

      animeTitle
      animeType
      animeNumEpisode
      releaseDate
      endDate
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      animeDescription
      animeRelated
      animeCharacters
      animeVoices
      animeStaff
      url

      3749
      NaN
      NaN
      0.0
      NaT
      NaT
      NaN
      None
      None
      None
      NaN
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/10079/Hoshizora_...

      17127
      NaN
      NaN
      0.0
      NaT
      NaT
      NaN
      None
      None
      None
      NaN
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/4379/Inmu__Ikeni...

```
# number duplicated
df[df.duplicated()].shape
```

    (2008, 16)

```
# drop all line duplicated
df.drop_duplicates(inplace=True)
```

```
# as we can see, we eliminated all duplicated line
df[df.duplicated()]
```

      animeTitle
      animeType
      animeNumEpisode
      releaseDate
      endDate
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      animeDescription
      animeRelated
      animeCharacters
      animeVoices
      animeStaff
      url

```
#we see the number of null value for each columns
df.isnull().sum()
```

    animeTitle              2
    animeType               2
    animeNumEpisode       509
    releaseDate           368
    endDate             10577
    animeNumMembers         2
    animeScore           5690
    animeUsers           5690
    animeRank            1823
    animePopularity         2
    animeDescription        8
    animeRelated            0
    animeCharacters         0
    animeVoices             0
    animeStaff              0
    url                     0
    dtype: int64

```
df['animeNumEpisode'].fillna(-1,inplace=True)
df['endDate'].fillna(-1,inplace=True)
df['releaseDate'].fillna(-1,inplace=True)
```

```
# we check the only two line null of description column
df[df.animeDescription.isnull()]
```

      animeTitle
      animeType
      animeNumEpisode
      releaseDate
      endDate
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      animeDescription
      animeRelated
      animeCharacters
      animeVoices
      animeStaff
      url

      3749
      NaN
      NaN
      0.0
      NaT
      NaT
      NaN
      None
      None
      None
      NaN
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/10079/Hoshizora_...

      7705
      Yuru Camp△ Movie
      Movie
      1.0
      2022-01-01
      NaT
      33071.0
      None
      None
      None
      3163.0
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/38475/Yuru_Camp△...

      7717
      Argonavis from BanG Dream! Movie
      Movie
      1.0
      2022-01-01
      NaT
      1105.0
      None
      None
      None
      10933.0
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/45731/Argonavis_...

      9215
      Kaeru no Pickles: Kimochi no Iro
      TV
      12.0
      2020-10-04
      2020-10-04
      777.0
      5.54
      117
      10253
      11805.0
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/42717/Kaeru_no_P...

      11876
      Mayi He Da Xiang
      Movie
      1.0
      1987-01-01
      NaT
      15.0
      None
      None
      17242
      18926.0
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/43616/Mayi_He_Da...

      17127
      NaN
      NaN
      0.0
      NaT
      NaT
      NaN
      None
      None
      None
      NaN
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/4379/Inmu__Ikeni...

      17589
      Kuma no Gakkou: Patissier Jackie to Ohisama no...
      Movie
      1.0
      2017-08-25
      NaT
      223.0
      None
      None
      16938
      15665.0
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/34755/Kuma_no_Ga...

      18722
      Yukai na Animal Bus
      TV
      12.0
      2017-01-08
      2017-01-08
      240.0
      None
      None
      14533
      15442.0
      NaN
      ['Haha wo Tazunete Sanzenri', 'Marco: Haha wo ...
      []
      []
      [['De Amicis, Edmondo', 'Original Creator']]
      https://myanimelist.net/anime/34794/Yukai_na_A...

```
# we drop the null column in animeDescription
df.dropna(subset=['animeDescription'], inplace=True)
```

```
# check null
df[df.animeDescription.isnull()].head()
```

      animeTitle
      animeType
      animeNumEpisode
      releaseDate
      endDate
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      animeDescription
      animeRelated
      animeCharacters
      animeVoices
      animeStaff
      url

```
# we drop the null column in endDate
df.dropna(subset=['endDate'], inplace=True)
```

```
# as we can see, we just need animeScore, animeUsers,animeRank to fix
df.isnull().sum()
```

    animeTitle             0
    animeType              0
    animeNumEpisode        0
    releaseDate            0
    endDate                0
    animeNumMembers        0
    animeScore          5683
    animeUsers          5683
    animeRank           1819
    animePopularity        0
    animeDescription       0
    animeRelated           0
    animeCharacters        0
    animeVoices            0
    animeStaff             0
    url                    0
    dtype: int64

```
#check if there are negative scores
df[df.animeScore <= 0]
```

      animeTitle
      animeType
      animeNumEpisode
      releaseDate
      endDate
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      animeDescription
      animeRelated
      animeCharacters
      animeVoices
      animeStaff
      url

```
##check if there are negative animeUsers
df[df.animeUsers<= 0]
```

      animeTitle
      animeType
      animeNumEpisode
      releaseDate
      endDate
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      animeDescription
      animeRelated
      animeCharacters
      animeVoices
      animeStaff
      url

```
#check if there are negative animeRank
df[df.animeRank<= 0]
```

      animeTitle
      animeType
      animeNumEpisode
      releaseDate
      endDate
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      animeDescription
      animeRelated
      animeCharacters
      animeVoices
      animeStaff
      url

```
#we replace -1 in place of null, because we believe that the number of rows is large to eliminate
df.fillna(value=-1, inplace=True)
```

```
# as we can see, we fix all column
df.isnull().sum()
```

    animeTitle          0
    animeType           0
    animeNumEpisode     0
    releaseDate         0
    endDate             0
    animeNumMembers     0
    animeScore          0
    animeUsers          0
    animeRank           0
    animePopularity     0
    animeDescription    0
    animeRelated        0
    animeCharacters     0
    animeVoices         0
    animeStaff          0
    url                 0
    dtype: int64

```
# reset index
df.reset_index(drop=True, inplace=True)
```

```
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 19120 entries, 0 to 19119
    Data columns (total 16 columns):
     #   Column            Non-Null Count  Dtype
    ---  ------            --------------  -----
     0   animeTitle        19120 non-null  object
     1   animeType         19120 non-null  object
     2   animeNumEpisode   19120 non-null  float64
     3   releaseDate       19120 non-null  object
     4   endDate           19120 non-null  object
     5   animeNumMembers   19120 non-null  float64
     6   animeScore        19120 non-null  float64
     7   animeUsers        19120 non-null  float64
     8   animeRank         19120 non-null  float64
     9   animePopularity   19120 non-null  float64
     10  animeDescription  19120 non-null  object
     11  animeRelated      19120 non-null  object
     12  animeCharacters   19120 non-null  object
     13  animeVoices       19120 non-null  object
     14  animeStaff        19120 non-null  object
     15  url               19120 non-null  object
    dtypes: float64(6), object(10)
    memory usage: 2.3+ MB

# Delete Stopwords & Stem & punctuation

```
# function to delete stopwords, punctuation adn stem
def filter_sentence(description):
  stop_words = set(stopwords.words('english')) # set of stopwords
  nl = WordNetLemmatizer()
  description = description.translate(str.maketrans('', '', string.punctuation)) # delete punctuation
  word_tokens = word_tokenize(description) # return list of word, very similar to split
  filtered_sentence = [nl.lemmatize(w).lower() for w in word_tokens if not w.lower() in stop_words] # delete stopwords
  filtered_description = " ".join(filtered_sentence)
  return filtered_description
```

```
df["animeDescription"] = df["animeDescription"].apply(filter_sentence) #apply the function
```

```
df['animeDescription'].head()
```

    0    story follows racoonlike tanuki creature named...
    1    synopsis information added title help improve ...
    2    animated promotional video bundled release ido...
    3    synopsis information added title help improve ...
    4         fire safety anime directed tsutomu shibayama
    Name: animeDescription, dtype: object

#REQ 2

## REQ 2.1

```
## return all words contians in the specified column
def retrieve_word_column(column):
    huge_list = []
    for i in range(len(df)):
        huge_list.extend(word_tokenize(df[column][i]))
    all_words = set(huge_list)
    return all_words
```

```
# create 2 dictionary: the first one contains word to index, and the second one
def retrieve_word_index(all_words):
  word_to_idx = dict()
  idx_to_word = dict()
  for idx, value in enumerate(all_words):
    word_to_idx[value] = idx
    idx_to_word[idx] = value
  return word_to_idx, idx_to_word
```

```
# function to add index in dictionary
# if word(index) not in dictionary added it with the corresponded document, else add document in word
def index_to_doc(index,dict_index,document):
  if index not in dict_index:
    dict_index[index] = [document]
  else:
    dict_index[index].append(document)
  return dict_index
```

```
# create dictionary which contain word to all documents which contain this word
def invert_index(column, word_to_idx):
    idx_to_documents = dict()
    for i,document in enumerate(df[column]):
      word_tokens = word_tokenize(document)
      word_tokens = set(word_tokens)
      for word in word_tokens:
        index = word_to_idx[word]
        idx_to_documents = index_to_doc(index,idx_to_documents,i)
    return idx_to_documents
```

```
# return the documents that contians the query
def match_query(df, query, column, word_to_idx, idx_to_documents):
  documents = []
  for word in query.split():
    if word in word_to_idx:
      documents.append(idx_to_documents[word_to_idx[word.strip().lower()]])
    else:
      return set() # if word not present in dictionary that contians all words, exit beacause the query given in input must match all words
  document = set.intersection(*map(set, documents))
  return df[["animeTitle","animeDescription","url"]].loc[list(document)]
```

```
column = "animeDescription"
all_words = retrieve_word_column(column)
word_to_idx,idx_to_word =  retrieve_word_index(all_words)
idx_to_documents = invert_index(column, word_to_idx)
match_query(df,"saiyan race","animeDescription",word_to_idx, idx_to_documents).head()
```

      animeTitle
      animeDescription
      url

      11611
      Dragon Ball Super: Broly
      fortyone year ago planet vegeta home infamous ...
      https://myanimelist.net/anime/36946/Dragon_Bal...

      445
      Dragon Ball Z Special 1: Tatta Hitori no Saish...
      bardock son gokus father lowranking saiyan sol...
      https://myanimelist.net/anime/986/Dragon_Ball_...

      13031
      Dragon Ball Z
      five year winning world martial arts tournamen...
      https://myanimelist.net/anime/813/Dragon_Ball_Z

```
# create 3 file Json which contain information
with open('word_to_idx.json', 'w') as fp:
    json.dump(word_to_idx, fp)

with open('idx_to_documents.json', 'w') as fp:
    json.dump(idx_to_documents, fp)

with open('idx_to_word.json', 'w') as fp:
    json.dump(idx_to_word, fp)
```

## REQ 2.2

```
#returns the updated dictionary with the value given in input
def index_to_doc_tfIdf(index ,dict_index, document, score):
  if index not in dict_index:
    dict_index[index] = [(document,score)]
  else:
    dict_index[index].append((document,score))
  return dict_index
```

```
#returns the updated dictionary with the value given in input
def doc_to_index_tfIdf(dict_tfIdf, document, score):
  if document not in dict_tfIdf:
    dict_tfIdf[document] = [score]
  else:
    dict_tfIdf[document].append(score)
  return dict_tfIdf
```

```
def get_number_occurrences(tokens):
    result = dict()
    for word in tokens:
      result[word] = result.get(word, 0) + 1 # if not in dictionary set 0, +1 otherwise
    return result
```

```
# function to calculate TF_IDF
def retrieve_TF_IDF_document(doc, word_to_idx, idx_to_documents):
  idx_doc_tfIdf = dict()
  idf_dict = dict()
  doc_tfIdf = dict()
  N = df.shape[0]

  for i, description in enumerate(doc):
    word_tokens = word_tokenize(description) # list of word
    num_occurences = get_number_occurrences(word_tokens) #dictionary that contains the number of occurancy for each word

    for word in num_occurences:
      index = word_to_idx[word.strip().lower()]
      word_occurance = len(idx_to_documents[index])

      # this block is useful to create a dictionary that contains for each word, the correspond idf
      if word in idf_dict:
        idf = idf_dict[word]
      else:
        idf = math.log(N / word_occurance) + 1
        idf_dict[word] = idf

      # calculate the tf
      tf = num_occurences[word] / len(word_tokens)
      score = tf * idf

      idx_doc_tfIdf = index_to_doc_tfIdf(index, idx_doc_tfIdf, i, score)
      doc_tfIdf = doc_to_index_tfIdf(doc_tfIdf,i,score)
  return idx_doc_tfIdf, idf_dict, doc_tfIdf
```

```
idx_doc_tfIdf, idf_dict, doc_index_tf_idf = retrieve_TF_IDF_document(df['animeDescription'], word_to_idx, idx_to_documents)
```

1. Calculate TFIDF of Query.
2. Retrieve the intersected documents.
3. Loop through each document and calculate the following.

```
# calculate the cosine similarity and return two valued, the first one a sorted list that contains the index of top document
# the second one the correspond dictionary, where the keys are the documents and the attribute the correspond similarity
import numpy as np
def cosine_similarity(query, column, df, word_to_idx, idx_to_documents, idx_doc_tfIdf, idf_dict, doc_index_tf_idf):
  query_terms = word_tokenize(query)
  results = dict()
  q = []
  match = match_query(df, query, column, word_to_idx, idx_to_documents)

  if len(match) == 0: return dict(),[]
  for query_term in query_terms:
    index = word_to_idx[query_term]
    postings_list = idx_doc_tfIdf[index]
    query_term_tfidf = compute_tfidf_query(query, query_term, idf_dict)
    q.append(query_term_tfidf)
    for doc in list(match.index):
      for doc_tf_idf, tfidf in postings_list:
        if doc == doc_tf_idf:
          results[doc] = results.get(doc, 0) + query_term_tfidf * tfidf
  for doc in results:
    results[doc] = results[doc] / (np.linalg.norm(doc_index_tf_idf[doc]) * np.linalg.norm(q))
  return results, list(sorted(results.keys(), key=lambda x: results[x], reverse=True))
```

```
# compute the tfidf for the query
def compute_tfidf_query(query, term, idf_dict):
  query_terms = word_tokenize(query)
  words_occ = get_number_occurrences(query_terms)
  tf = words_occ[term] / len(query_terms)
  if term not in idf_dict:
    idf = 0
  else:
    idf = idf_dict[term]
  return tf * idf
```

```
# calculate the cosine similarity
def retrieve_cosine_similarity(query, column_name, df):
  query = filter_sentence(query)
  all_words = retrieve_word_column(column_name)
  word_to_idx, idx_to_word =  retrieve_word_index(all_words)
  idx_to_documents = invert_index(column_name, word_to_idx)
  idx_doc_tfIdf, idf_dict, doc_index_tf_idf = retrieve_TF_IDF_document(df[column_name], word_to_idx, idx_to_documents)
  result = cosine_similarity(query, column_name, df, word_to_idx, idx_to_documents, idx_doc_tfIdf, idf_dict, doc_index_tf_idf)
  return result
```

```
query = 'saiyan race'
result_query_description = retrieve_cosine_similarity(query, "animeDescription", df)
cosine_result = pd.concat([df.loc[result_query_description[1]],pd.Series(sorted(result_query_description[0].values(),reverse=True),\
                                                               index=result_query_description[1], dtype="float32").rename('descriptionSimilarity')], axis=1)
cosine_result[['animeTitle', 'animeNumMembers', 'animeScore', 'animeUsers', 'animeRank', 'animePopularity','descriptionSimilarity']]
```

      animeTitle
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      descriptionSimilarity

      445
      Dragon Ball Z Special 1: Tatta Hitori no Saish...
      87614.0
      7.55
      56578.0
      1468.0
      1786.0
      0.302161

      11611
      Dragon Ball Super: Broly
      278607.0
      8.12
      188367.0
      402.0
      631.0
      0.186803

      13031
      Dragon Ball Z
      1033068.0
      8.15
      697557.0
      366.0
      95.0
      0.135501

```
df['animeDescription'].shape
```

    (19120,)

#REQ 3

```
df['animeTitleLower'] =  df["animeTitle"].apply(filter_sentence)
```

```
list_columns = ['animeTitle', 'animeType', 'animeNumEpisode', 'releaseDate', 'endDate', 'animeNumMembers', 'animeScore', 'animeUsers', 'animeRank', 'animePopularity', 'animeDescription', \
                'animeRelated', 'animeCharacters', 'animeVoices', 'animeStaff', 'url','animeTitleLower']
```

```
def find_match_req3(query, list_column, df):
  list_column.extend(["descriptionSimilarity","titleSimilarity"])
  column_normalize_bool = {"descriptionSimilarity":False, "titleSimilarity": False, "animeScore": False, "animeRank":True, "animePopularity": True, "animeNumMembers":False}
  column_weight = {"descriptionSimilarity": 0.2, "titleSimilarity": 0.2, "animeScore":0.2 ,"animeRank":0.2, "animePopularity": 0.05, "animeNumMembers":0.05}
  calculate_weight = recalculate_weight(column_weight, list_column,6)
  print(calculate_weight)
  # cosine similarity for title
  result_query_title = retrieve_cosine_similarity(query, "animeTitleLower", df)

  # cosine similarity for description
  result_query_description = retrieve_cosine_similarity(query, "animeDescription", df)

  #we take the corresponding lines from the data frame
  cosine_result_1 = pd.concat([df.loc[result_query_title[1]],pd.Series(sorted(result_query_title[0].values(),reverse=True),\
                                                               index=result_query_title[1], dtype="float32").rename('titleSimilarity')], axis=1)

  #we take the corresponding lines from the data frame
  cosine_result_2 = pd.concat([df.loc[result_query_description[1]],pd.Series(sorted(result_query_description[0].values(),reverse=True),\
                                                               index=result_query_description[1], dtype="float32").rename('descriptionSimilarity')], axis=1)

  # join the two dataframe (list_columns is present in the previous cell)
  cosine_result = pd.merge(cosine_result_1, cosine_result_2,  how='outer', on=list_columns)

  # fill the null value with zero
  cosine_result = cosine_result.fillna(0)

  #when we go to match, it is likely that one of the two is not present, so we will not create the table with zero as default value
  if "titleSimilarity" not in cosine_result.columns: cosine_result["titleSimilarity"] = 0
  if "descriptionSimilarity" not in cosine_result.columns: cosine_result["descriptionSimilarity"] = 0

  # column which contains the new similarity
  cosine_result['newSimilarity'] = 0
  for column in list_column:
    small_important = column_normalize_bool[column] #True or False
    value = normalize_column(cosine_result,column, calculate_weight[column],small_important) # calculate the weight of the column
    cosine_result['newSimilarity'] += value #added value in the newSimilarity

  return  cosine_result.sort_values(by='newSimilarity',ascending=False)

```

```
#recalculate the weight based on the user's choice
def recalculate_weight(diz_orig_weight, list_columns, n):
  if n == len(diz_orig_weight): return diz_orig_weight
  sum = 0
  diz_new_weight = dict()
  for column in list_columns:
    sum += diz_orig_weight[column]
  for column in list_columns:
    diz_new_weight[column] = diz_orig_weight[column] / sum
  return diz_new_weight
```

```
# function useful to normalize the value
#small_important is a boolean, True it means that the smaller the value, the more important that value is, so we have to do `1 - value` to normalize
def normalize_column(cosine_result,column_name, weight,small_important):
  max = cosine_result[column_name].max()
  if max == 0: return 0
  #print(column_name, (1 - cosine_result[cosine_result.animeTitle == 'Dragon Ball'][column_name] / max * weight).iloc[0])
  if small_important: return (1- cosine_result[column_name] / max) * weight
  return (cosine_result[column_name] / max) * weight
```

## Explain algorithm
To calculate the similarity, we don't use just the description, but we add some other factors, each one with its own weight. We use the `titleSimilarity`, `animeScore`, `animeRank`, `animePopularity` and `animeNumMembers`.

We believe that the `titleSimilarity` is the most important factor, so it has the biggest weight.

The users can choose the columns used to calculate the similarity, the only two that are mandatory are `titleSimilarity` and `descriptionSimilarity`, instead the other four factors can be included or not in the computation of the new similarity.
When the users don`t choose all the columns, the weights are automatically updated with a proportion.

For example, we can have the following:
$$0.3:0.7=x:1$$
Where 0.3 is the weight of that column that we want to update and 0.7 is the sum of the weights of all the columns chosen.

Given the new weights and the list of columns $C$ chosen by the user, we are going to compute the new similarity, using the following formula:

$$\text{new_similarity} = \sum_c \frac{v_c}{max_c} \cdot{w_c} \quad \forall c \in C$$

where $v_c$ is the cosine similarity when $c$ is the `animeTitle` or `animeDescription` column, while it is the value of the column itself for all the other columns. Instead, $max_c$ is the maximum value between all the values of the column and $w_c$ is the weights associated to the column $c$.

## TEST 1

Now, we will show an example of the previous similarity. If we use the query `naruto` and request the new similarity to be computed also with the values in the `animeRank` and `animeScore` columns, we will do the following steps.

First of all, the weights have assigned a default values to each of them:
*   descriptionSimilarity $\rightarrow$ 0.2
*   titleSimilarity $\rightarrow$ 0.2
*   animeScore $\rightarrow$ 0.2
*   animeRank $\rightarrow$ 0.2
*   animePopularity $\rightarrow$ 0.05
*   animeNumMembers $\rightarrow$ 0.05

So, as we said, initially we update the weights just for the columns selected by the user, in this case we have:
*   descriptionSimilarity $\rightarrow$ 0.25
*   titleSimilarity $\rightarrow$ 0.25
*   animeRank $\rightarrow$ 0.25
*   animeScore $\rightarrow$ 0.25

In the first cell below, we can find the ranking obtained with the similarity computed as before, while in the second cell we can see the new ranking obtained with the custom similarity.

As we can see, the first element in the first ranking doesn't really match  what we were looking for. While if we see the second ranking, given by our algorithm, we can see that all the results obtained have Naruto in the title
since we considered also the title, rank and score, instead of using just the description like in the first ranking.

```
query = 'naruto'
result_query_description = retrieve_cosine_similarity(query, "animeDescription", df)
cosine_result = pd.concat([df.loc[result_query_description[1]],pd.Series(sorted(result_query_description[0].values(),reverse=True),\
                                                               index=result_query_description[1], dtype="float32").rename('descriptionSimilarity')], axis=1)
cosine_result[['animeTitle', 'animeNumMembers', 'animeScore', 'animeUsers', 'animeRank', 'animePopularity','descriptionSimilarity']].head()
```

      animeTitle
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      descriptionSimilarity

      4437
      Juliet
      1059.0
      4.73
      394.0
      11724.0
      11012.0
      0.463412

      15692
      Boruto: Naruto the Movie - Naruto ga Hokage ni...
      90761.0
      7.37
      50673.0
      2093.0
      1759.0
      0.408632

      7299
      Naruto: Shippuuden
      1929365.0
      8.22
      1125164.0
      299.0
      16.0
      0.385227

      15721
      Naruto: Shippuuden - Shippuu! "Konoha Gakuen" Den
      58260.0
      7.15
      28567.0
      3103.0
      2290.0
      0.376075

      14150
      Naruto: Shippuuden Movie 2 - Kizuna
      219713.0
      7.28
      127336.0
      2506.0
      807.0
      0.371825

```
query = 'naruto'
find = find_match_req3(query, ['animeRank','animeScore'], df)
find[['animeTitle','animeNumMembers', 'animeScore', 'animeUsers', 'animeRank', 'animePopularity',"titleSimilarity","descriptionSimilarity",'newSimilarity']].head()
```

    {'animeRank': 0.25, 'animeScore': 0.25, 'descriptionSimilarity': 0.25, 'titleSimilarity': 0.25}

      animeTitle
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      titleSimilarity
      descriptionSimilarity
      newSimilarity

      0
      Naruto
      2292958.0
      7.95
      1512406.0
      609.0
      8.0
      1.000000
      0.333702
      0.909617

      1
      Naruto: Shippuuden
      1929365.0
      8.22
      1125164.0
      299.0
      16.0
      0.661635
      0.385227
      0.867242

      3
      Boruto: Naruto the Movie - Naruto ga Hokage ni...
      90761.0
      7.37
      50673.0
      2093.0
      1759.0
      0.609035
      0.408632
      0.804941

      12
      Naruto: Shippuuden Movie 6 - Road to Ninja
      274002.0
      7.67
      166060.0
      1124.0
      633.0
      0.420373
      0.346186
      0.752616

      9
      Naruto: Shippuuden Movie 2 - Kizuna
      219713.0
      7.28
      127336.0
      2506.0
      807.0
      0.496647
      0.371825
      0.745980

## TEST 2

We did another test with a different query `goku`, and we inserted all columns (`animeNumMembers`, `animeScore`, `animeRank`, `animePopularity`). So, the weights remained the same one as defined by default.

As we can see, in the first cell, we have that the best match has a negative value for the score, while our algorithm, since it uses the score value to compute the similarity, it will not rank that anime so high. In fact, we don't have it in the top-5 at all.

```
query = 'goku'
result_query_description = retrieve_cosine_similarity(query, "animeDescription", df)
cosine_result = pd.concat([df.loc[result_query_description[1]],pd.Series(sorted(result_query_description[0].values(),reverse=True),\
                                                               index=result_query_description[1], dtype="float32").rename('descriptionSimilarity')], axis=1)
cosine_result[['animeTitle','animeNumMembers', 'animeScore', 'animeUsers', 'animeRank', 'animePopularity','descriptionSimilarity']].head()
```

      animeTitle
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      descriptionSimilarity

      1915
      Son O-gong gwa Byeoldeul-ui Jeonjaeng
      263.0
      -1.00
      -1.0
      13448.0
      15053.0
      0.621845

      4116
      Dragon Ball Movie 4: Saikyou e no Michi
      51628.0
      7.13
      27582.0
      3225.0
      2465.0
      0.604225

      3252
      Dragon Ball Z: Summer Vacation Special
      11351.0
      6.59
      4477.0
      5666.0
      5058.0
      0.503414

      5030
      Dragon Ball Z Movie 07: Kyokugen Battle!! Sand...
      103961.0
      6.85
      64806.0
      4410.0
      1572.0
      0.462226

      7730
      Dragon Ball Z Movie 07: Kyokugen Battle!! Sand...
      103961.0
      6.85
      64806.0
      4409.0
      1572.0
      0.462226

```
query = 'goku'
find = find_match_req3(query, ['animeNumMembers', 'animeScore', 'animeRank', 'animePopularity'], df)
find[['animeTitle','animeNumMembers', 'animeScore', 'animeUsers', 'animeRank', 'animePopularity',"titleSimilarity","descriptionSimilarity",'newSimilarity']].head()
```

    {'descriptionSimilarity': 0.2, 'titleSimilarity': 0.2, 'animeScore': 0.2, 'animeRank': 0.2, 'animePopularity': 0.05, 'animeNumMembers': 0.05}

      animeTitle
      animeNumMembers
      animeScore
      animeUsers
      animeRank
      animePopularity
      titleSimilarity
      descriptionSimilarity
      newSimilarity

      0
      Goku Sayonara Zetsubou Sensei
      59619.0
      7.84
      28392.0
      779.0
      2265.0
      0.607027
      0.000000
      0.627471

      3
      Dragon Ball Movie 4: Saikyou e no Michi
      51628.0
      7.13
      27582.0
      3225.0
      2465.0
      0.000000
      0.604225
      0.566872

      1
      Dragon Ball Kai: Mirai ni Heiwa wo! Goku no Ta...
      41082.0
      7.52
      23051.0
      1558.0
      2766.0
      0.395872
      0.000000
      0.536077

      32
      Kill la Kill
      1462057.0
      8.06
      777422.0
      478.0
      47.0
      0.000000
      0.084141
      0.519797

      6
      Dragon Ball Z Movie 07: Kyokugen Battle!! Sand...
      103961.0
      6.85
      64806.0
      4409.0
      1572.0
      0.000000
      0.462226
      0.501401

# REQ 4

In this requirement we have performed a sentimental analysis using the vader model. The vader model is very simple, it takes a string as input and is able to understand if the content of the string is positive or negative. It is essentially based on keywords like hate, sad for negative comments, or love for positive comments.
Using the polarity_scores function, it returns a dictionary containing 4 values:
- negative
- neutral
- positive
- compound

the sum between negative + neutral + positive must be 1, and each of those indicate how much the comment is positive, neutral or negative. The compound, on the other hand, is the normalized variable that we are going to use.

For this analysis we can't delete the stopwords, because the set of stopwords contains some words like `no`, `not` etc, and this words are very important to understand the sentiment of comment.

When I loaded the HTML file, I only uploaded the first 4 reviews because to get the first 5 I had to make another request to the site that was in another url, so unfortunately the analysis will be carried out on 4 comments.

```
col=["Reviews", "url"]
reviews_df = pd.read_csv("/content/drive/MyDrive/ADM/HM3/reviews_tsv.tsv",sep='\t', names=col)
```

```
import ast
reviews_df.Reviews = reviews_df.Reviews.apply(ast.literal_eval)
```

```
reviews_df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 19128 entries, 0 to 19127
    Data columns (total 2 columns):
     #   Column   Non-Null Count  Dtype
    ---  ------   --------------  -----
     0   Reviews  19128 non-null  object
     1   url      19128 non-null  object
    dtypes: object(2)
    memory usage: 299.0+ KB

```
reviews_df.Reviews
```

    0                                                       []
    1                                                       []
    2                                                       []
    3                                                       []
    4                                                       []
                                   ...
    19123    [~Multilingual review, English & Español: The ...
    19124    [Haikyuu: To the Top, or the 4th installment t...
    19125    [Ping Pong The Animation is an exceptional sho...
    19126    [*This review contains very minor spoilers*  D...
    19127    [Going back to One Punch Man, I really thought...
    Name: Reviews, Length: 19128, dtype: object

```
reviews_df[reviews_df.Reviews.str.len() == 0]
```

      Reviews
      url

      0
      []
      https://myanimelist.net/anime/11673/Hal_no_Fue

      1
      []
      https://myanimelist.net/anime/45687/Hailuo_Wan

      2
      []
      https://myanimelist.net/anime/50066/Hajimari_w...

      3
      []
      https://myanimelist.net/anime/49899/Hajimari_n...

      4
      []
      https://myanimelist.net/anime/39161/Habatake_P...

      ...
      ...
      ...

      19056
      []
      https://myanimelist.net/anime/36566/Crayon_Shi...

      19064
      []
      https://myanimelist.net/anime/40291/Lost_One_n...

      19068
      []
      https://myanimelist.net/anime/10947/Koukaku_Ki...

      19075
      []
      https://myanimelist.net/anime/34178/Inazuma_El...

      19077
      []
      https://myanimelist.net/anime/18053/Koi_to_Sen...

8711 rows × 2 columns

```
reviews_df[['review1','review2',"review3","review4"]] = pd.DataFrame(reviews_df.Reviews.tolist(), index= reviews_df.index)
```

```
import nltk
nltk.download('vader_lexicon')
from nltk.sentiment.vader import SentimentIntensityAnalyzer

sid = SentimentIntensityAnalyzer()
```

    [nltk_data] Downloading package vader_lexicon to /root/nltk_data...
    [nltk_data]   Package vader_lexicon is already up-to-date!

```
reviews_df['review1'].head()
```

    0    None
    1    None
    2    None
    3    None
    4    None
    Name: review1, dtype: object

```
reviews_df.fillna(0,inplace=True)
```

```
cols = ['review1', 'review2', 'review3', 'review4']
for col in cols:
  reviews_df[col] = reviews_df[col].apply(lambda review: (sid.polarity_scores(review)['compound']) if review != 0 else review)
```

```
reviews_df[reviews_df.review1 == 0]['review1']
```

    0        0.0
    1        0.0
    2        0.0
    3        0.0
    4        0.0
            ...
    19056    0.0
    19064    0.0
    19068    0.0
    19075    0.0
    19077    0.0
    Name: review1, Length: 8772, dtype: float64

```
reviews_df['result'] = (reviews_df['review1'] + reviews_df['review2'] + reviews_df['review3'] + reviews_df['review4']) / 4
```

```
df = df.drop_duplicates(subset='url', keep="last")
```

```
reviews_df = reviews_df.drop_duplicates(subset='url', keep="last")
```

```
df_temp = pd.merge(df, reviews_df,  how='inner', on="url")
```

## REQ 4.1

```
df_temp.groupby(by='animeType')['result'].mean().sort_values(ascending=False)
```

    animeType
    TV         0.245996
    Special    0.146842
    OVA        0.134859
    Movie      0.128069
    ONA        0.091577
    Music      0.030172
    Unknown    0.000000
    Name: result, dtype: float64

As we can see, the best animeType is TV

##REQ 4.2

```
df_temp.plot(kind="scatter", figsize=(10,5), y ='animeNumEpisode',x='result')
```

As we can see in the plot, more the number of episodes increases, more likely the comments will be neutral.

## REQ 4.3

```
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize=(8, 6))
sns.heatmap(df_temp[['animeScore','result']].corr(method='pearson'), annot=True)
plt.show()
```

As we can see, from the matrix of correlation, between `animeScore` and `result` there is slight positive correlation. So if one grows, also the other in part grows.

#REQ 5 THEORY

$L$ is the list of appointments given in input, where each element correspond to the number of minutes of the appointment.

$T$ is the new list, where in the last element we will find the sum of the maximum number of minutes that the personal trainer can do.

$i$ represents the $i$-th appointment in the list.

The idea is that when we fill the array with the sum of the appointments, the first one that we put in is given by the fact that it is the first that matches. Moreover, when we arrive at the second appointment, we choose the maximum between the first and the second appointment because we can't have two consecutive appointment.

From the third we change methodology, because now we can take more than one appointment, so we will do: on $T[i]$ (in this case $i=3$) we will put the maximum between the previous($T[i-1]$) and the sum of $T[i] + T[i-2]$ (in this case we will put the maximum between the second appointment and the sum between the first and the third one). Continuing like this we will find in the last element the maximum number of minutes that a personal trainer could perform.

Below the cell with the pseudocode, and then the cell with the function `find_max` that return the list $T$.

Below we also show, how from the vector $T$, we can extrapolate the appointments that we have chosen, that is the optimal solution.

\begin{align}
        \text{T}(i) = \left\{
        \begin{array}{cl}
        l[0]& i = 0 \\
        max(l[i], l[i-1]) & i = 1 \\
        max(l[i-1],l[i-2] + l[i]) & otherwise
        \end{array}
        \right.
    \end{align}

```
lista=[30,40,25,50,30,20]
def find_max(lista):
    if len(lista) == 1: # i == 0
        return lista[0]
    elif len(lista) == 2: # i == 1
        max(lista[1],lista[0])
    result = [x for x in lista]
    for i in range(2,len(lista)): # otherwise
        result[i] = max(result[i-1],result[i-2] + result[i])
    return result
print(find_max(lista))
```

    [30, 40, 55, 90, 90, 110]

Now that we have the vector of the maximum sum of minutes for a personal trainer, we will go to extrapolate the appointments chosen. Here the variabile that we need:
- $c$, last element from the list of the appointments, so the sum of all the optimal appointments;
- $i$, the index that we use on the while to scroll backwards the list $T$, and it will be initialized as the index of the last element.

We start doing a backwards scroll from the list of the appointments $T$ to understand what we chosen.
The idea is based on the fact that if the last element is bigger than the penultimate, it means that the last appointment has been chosen, so we can add it to our list. Furthermore, we subtract from $c$ the number of minutes of the appointment chosen (we need this variable to understand if we choose the first element on the list of appointments or not) then we decrease of 2 positions our index $i$ because we know that the penultimate appointment wasn't chosen.
Consequentially, we arrive at the third last appointment and we repeat the same process to the end of the list.
If we are at the first element and $c > 0$, it means that we had chosen also the first element so we add it to the list of the appointments chosen.

```
def print_choises(lista_orginal, list_sum_appointment):
    result = []
    i = len(lista_orginal) -1
    c = list_sum_appointment[-1] # the sum of all appointment choises
    while(c > 0 and i > -1):
        if list_sum_appointment[i] > list_sum_appointment[i-1]:
            result.append(lista_orginal[i])
            c -= lista_orginal[i]
            i-=2
        elif i == 0:
            result.append(lista_orginal[i])
            c -= lista_orginal[0]
    result.reverse()
    return result

print_choises(lista, find_max(lista))
```

    [40, 50, 20]