# Information

## RAM Info

```python
!cat /proc/meminfo
```

    MemTotal:       13335276 kB
    MemFree:         4100312 kB
    MemAvailable:   11245716 kB
    Buffers:           99588 kB
    Cached:          6936900 kB
    SwapCached:            0 kB
    Active:          1691180 kB
    Inactive:        6867788 kB
    Active(anon):    1370056 kB
    Inactive(anon):    12648 kB
    Active(file):     321124 kB
    Inactive(file):  6855140 kB
    Unevictable:           0 kB
    Mlocked:               0 kB
    SwapTotal:             0 kB
    SwapFree:              0 kB
    Dirty:               308 kB
    Writeback:             4 kB
    AnonPages:       1522444 kB
    Mapped:           983000 kB
    Shmem:             13204 kB
    Slab:             342624 kB
    SReclaimable:     294580 kB
    SUnreclaim:        48044 kB
    KernelStack:        4656 kB
    PageTables:        12836 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:     6667636 kB
    Committed_AS:    3580880 kB
    VmallocTotal:   34359738367 kB
    VmallocUsed:           0 kB
    VmallocChunk:          0 kB
    AnonHugePages:         0 kB
    ShmemHugePages:        0 kB
    ShmemPmdMapped:        0 kB
    HugePages_Total:       0
    HugePages_Free:        0
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    DirectMap4k:      280564 kB
    DirectMap2M:     7059456 kB
    DirectMap1G:     8388608 kB

```python

```

## CPU Info

```python
!cat /proc/cpuinfo
```

    processor	: 0
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU @ 2.20GHz
    stepping	: 0
    microcode	: 0x1
    cpu MHz		: 2200.000
    cache size	: 56320 KB
    physical id	: 0
    siblings	: 2
    core id		: 0
    cpu cores	: 1
    apicid		: 0
    initial apicid	: 0
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 13
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat arch_capabilities
    bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
    bogomips	: 4400.00
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 46 bits physical, 48 bits virtual
    power management:

    processor	: 1
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU @ 2.20GHz
    stepping	: 0
    microcode	: 0x1
    cpu MHz		: 2200.000
    cache size	: 56320 KB
    physical id	: 0
    siblings	: 2
    core id		: 0
    cpu cores	: 1
    apicid		: 1
    initial apicid	: 1
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 13
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat arch_capabilities
    bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
    bogomips	: 4400.00
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 46 bits physical, 48 bits virtual
    power management:

# GPU Info

```python
!nvidia-smi
```

    Sat Jul 20 10:23:39 2019
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 418.67       Driver Version: 410.79       CUDA Version: 10.0     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
    | N/A   68C    P0    31W /  70W |    464MiB / 15079MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                       GPU Memory |
    |  GPU       PID   Type   Process name                             Usage      |
    |=============================================================================|
    +-----------------------------------------------------------------------------+

# Current Directory

```python
!pwd
```

    /content

# Download A File

```python
!wget http://nlp.stanford.edu/data/glove.6B.zip
```

    --2019-07-20 10:33:46--  http://nlp.stanford.edu/data/glove.6B.zip
    Resolving nlp.stanford.edu (nlp.stanford.edu)... 171.64.67.140
    Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:80... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://nlp.stanford.edu/data/glove.6B.zip [following]
    --2019-07-20 10:33:46--  https://nlp.stanford.edu/data/glove.6B.zip
    Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:443... connected.
    HTTP request sent, awaiting response... 301 Moved Permanently
    Location: http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip [following]
    --2019-07-20 10:33:47--  http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip
    Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22
    Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 862182613 (822M) [application/zip]
    Saving to: ‘glove.6B.zip’

    glove.6B.zip        100%[===================>] 822.24M  25.4MB/s    in 33s

    2019-07-20 10:34:20 (24.9 MB/s) - ‘glove.6B.zip’ saved [862182613/862182613]

# Git

```python
!rm -rf /content/docs
```

```python
!git clone https://github.com/tensorflow/docs.git
%cd docs
!ls

%cd
```

    Cloning into 'docs'...
    remote: Enumerating objects: 37, done.
    remote: Counting objects: 100% (37/37), done.
    remote: Compressing objects: 100% (25/25), done.
    remote: Total 86773 (delta 19), reused 21 (delta 12), pack-reused 86736
    Receiving objects: 100% (86773/86773), 64.93 MiB | 15.80 MiB/s, done.
    Resolving deltas: 100% (65423/65423), done.
    /root/docs
    AUTHORS  CODEOWNERS  CONTRIBUTING.md  LICENSE  README.md  setup.py  site  tools
    /root

# Markdown

#Heading 1
##Heading 2
###Heading 3
####Heading 4
#####Heading 5
######Heading 6

---

Paragraph
text `Inline Code` text
~~Mistaken text.~~
*Italics*
**Bold**

---

Tasks
- [ ] a task list item
- [ ] list syntax required
- [ ] normal **formatting**
- [ ] incomplete
- [x] completed

---

Code Blocks

    4 space indention
    makes full-width
    standard code blocks

```js
var now = new Date();

var days = new Array('Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday');

var months = new Array('January','February','March','April','May','June','July','August','September','October','November','December');

var date = ((now.getDate()<10) ? "0" : "")+ now.getDate();

function fourdigits(number)	{
	return (number < 1000) ? number + 1900 : number;
								}
today =  days[now.getDay()] + ", " +
         months[now.getMonth()] + " " +
         date + ", " +
         (fourdigits(now.getYear())) ;

document.write(today);
```

```css
#sc_drag_area {
  height:100px;
  left:150px;
  position: absolute;
  top:100px;
  width:250px;
  z-index: 9999;
}
```

---

* List item one
* List item two
    * A nested item

---

1. Number list item one
	1.1. A nested item
2. Number list item two
3. Number list item three

---

> Quote
>
> Second line Quote

---

Standard link =  http://ghost.org
[Custom Text Link](http://ghost.org)

---

---

# Table

| Left-Aligned  | Center Aligned  | Right Aligned |
| :------------ |:---------------:| -----:|
| col 3 is      | some wordy text | \$1600 |
| col 2 is      | centered        |   \$12 |
| zebra stripes | are neat        |    \$1 |

# Equations

$$
\begin{array}{c}
y_1 \\\
y_2 \mathtt{t}_i \\\
z_{3,4}
\end{array}
$$

$$
\begin{array}{c}
y_1 \cr
y_2 \mathtt{t}_i \cr
y_{3}
\end{array}
$$

$$\begin{eqnarray}
x' &=& &x \sin\phi &+& z \cos\phi \\
z' &=& - &x \cos\phi &+& z \sin\phi \\
\end{eqnarray}$$

$$
x=4
$$

\begin{equation}
E = F \cdot s
\end{equation}

# Forms

Forms provide an easy way to parameterize code. From a code cell, select **Insert → Add form field**.  When you change the value in a form, the corresponding value in the code will change.

```python
#@title String fields

text = 'value' #@param {type:"string"}
dropdown = '1st option' #@param ["1st option", "2nd option", "3rd option"]
text_and_dropdown = 'value' #@param ["1st option", "2nd option", "3rd option"] {allow-input: true}

print(text)
print(dropdown)
print(text_and_dropdown)
```

    value
    1st option
    value

```python
#@title Raw fields

raw_input = None #@param {type:"raw"}
raw_dropdown = raw_input #@param [1, "raw_input", "False", "'string'"] {type:"raw"}

print(raw_input)
print(raw_dropdown)
```

    None
    None

```python
#@title Date fields
date_input = '2018-03-22' #@param {type:"date"}

print(date_input)
```

    2018-03-22

```python
#@title Number fields
number_input = 10.0 #@param {type:"number"}
number_slider = 1 #@param {type:"slider", min:-1, max:1, step:0.1}

integer_input = 10 #@param {type:"integer"}
integer_slider = 0 #@param {type:"slider", min:0, max:100, step:1}

print(number_input)
print(number_slider)

print(integer_input)
print(integer_slider)
```

    10.0
    0
    10
    1

```python
#@title Boolean fields
boolean_checkbox = True #@param {type:"boolean"}
boolean_dropdown = True #@param ["False", "True"] {type:"raw"}

print(boolean_checkbox)
print(boolean_dropdown)
```

    True
    True

```python
#@title ## Markdown
#@markdown You can also include Markdown in forms.

#@markdown ---
#@markdown ### Enter a file path:
file_path = "" #@param {type:"string"}
#@markdown ---

```

# Hiding code

You can change the view of the form by selecting **Edit → Show/hide code** or using the toolbar above the selected code cell. You can see both code and the form, just the form, or just the code.

```python
#@title Double click on this line in the code cell. { display-mode: "form" }

option1 = 'A' #@param ["A", "B", "C"]
print('You selected', option1)

```

```python
#@title After running this cell manually, it will auto-run if you change the selected value. { run: "auto" }

option2 = "A" #@param ["A", "B", "C"]
print('You selected', option2)

```

# Mounting

```python
#@title ## Mounting Gdrive

USE_G_COLAB = True #@param {type:"boolean"}

if USE_G_COLAB:
    from google.colab import drive

    drive.mount('/content/drive', force_remount=True)
```

```python
#@title ## Project Root

root_dir = ''

if USE_G_COLAB:
    root_dir = '/content/drive/My Drive/workshops/2019_07_21/sessions_01/' #@param {type:"string"}
```


```
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive

```
# 패키지 설치
!pip install mxnet==1.6.0 gluonnlp==0.9.1 sentencepiece==0.1.91 pandas==1.0.5 transformers==2.11.0 pytorch_lightning==0.8.1 onnx onnxruntime coremltools
```

    Requirement already satisfied: mxnet==1.6.0 in /usr/local/lib/python3.6/dist-packages (1.6.0)
    Requirement already satisfied: gluonnlp==0.9.1 in /usr/local/lib/python3.6/dist-packages (0.9.1)
    Requirement already satisfied: sentencepiece==0.1.91 in /usr/local/lib/python3.6/dist-packages (0.1.91)
    Requirement already satisfied: pandas==1.0.5 in /usr/local/lib/python3.6/dist-packages (1.0.5)
    Requirement already satisfied: transformers==2.11.0 in /usr/local/lib/python3.6/dist-packages (2.11.0)
    Requirement already satisfied: pytorch_lightning==0.8.1 in /usr/local/lib/python3.6/dist-packages (0.8.1)
    Requirement already satisfied: onnx in /usr/local/lib/python3.6/dist-packages (1.8.0)
    Requirement already satisfied: onnxruntime in /usr/local/lib/python3.6/dist-packages (1.6.0)
    Collecting coremltools
      Downloading https://files.pythonhosted.org/packages/f1/30/32f5bce33571b00f8e43e290bb4a9561956bc7f35b96dcd48f300029b23f/coremltools-4.0-cp36-none-manylinux1_x86_64.whl (3.4MB)
         |████████████████████████████████| 3.4MB 4.3MB/s
    Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from mxnet==1.6.0) (0.8.4)
    Requirement already satisfied: requests<3,>=2.20.0 in /usr/local/lib/python3.6/dist-packages (from mxnet==1.6.0) (2.23.0)
    Requirement already satisfied: numpy<2.0.0,>1.16.0 in /usr/local/lib/python3.6/dist-packages (from mxnet==1.6.0) (1.19.4)
    Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from gluonnlp==0.9.1) (20.8)
    Requirement already satisfied: cython in /usr/local/lib/python3.6/dist-packages (from gluonnlp==0.9.1) (0.29.21)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas==1.0.5) (2018.9)
    Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas==1.0.5) (2.8.1)
    Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers==2.11.0) (3.0.12)
    Requirement already satisfied: tokenizers==0.7.0 in /usr/local/lib/python3.6/dist-packages (from transformers==2.11.0) (0.7.0)
    Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from transformers==2.11.0) (0.8)
    Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers==2.11.0) (0.0.43)
    Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers==2.11.0) (4.41.1)
    Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers==2.11.0) (2019.12.20)
    Requirement already satisfied: tensorboard>=1.14 in /usr/local/lib/python3.6/dist-packages (from pytorch_lightning==0.8.1) (2.4.0)
    Requirement already satisfied: PyYAML>=5.1 in /usr/local/lib/python3.6/dist-packages (from pytorch_lightning==0.8.1) (5.3.1)
    Requirement already satisfied: torch>=1.3 in /usr/local/lib/python3.6/dist-packages (from pytorch_lightning==0.8.1) (1.7.0+cu101)
    Requirement already satisfied: future>=0.17.1 in /usr/local/lib/python3.6/dist-packages (from pytorch_lightning==0.8.1) (0.18.2)
    Requirement already satisfied: protobuf in /usr/local/lib/python3.6/dist-packages (from onnx) (3.12.4)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from onnx) (1.15.0)
    Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.6/dist-packages (from onnx) (3.7.4.3)
    Requirement already satisfied: sympy in /usr/local/lib/python3.6/dist-packages (from coremltools) (1.1.1)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from coremltools) (1.4.1)
    Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from coremltools) (20.3.0)
    Collecting attr
      Downloading https://files.pythonhosted.org/packages/de/be/ddc7f84d4e087144472a38a373d3e319f51a6faf6e5fc1ae897173675f21/attr-0.3.1.tar.gz
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet==1.6.0) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet==1.6.0) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet==1.6.0) (2020.12.5)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet==1.6.0) (3.0.4)
    Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->gluonnlp==0.9.1) (2.4.7)
    Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.11.0) (1.0.0)
    Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.11.0) (7.1.2)
    Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (1.17.2)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (0.4.2)
    Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (51.0.0)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (1.0.1)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (1.7.0)
    Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (1.32.0)
    Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (0.10.0)
    Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (0.36.2)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->pytorch_lightning==0.8.1) (3.3.3)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy->coremltools) (1.1.0)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14->pytorch_lightning==0.8.1) (4.2.0)
    Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14->pytorch_lightning==0.8.1) (4.6)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14->pytorch_lightning==0.8.1) (0.2.8)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.14->pytorch_lightning==0.8.1) (1.3.0)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard>=1.14->pytorch_lightning==0.8.1) (3.3.0)
    Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<5,>=3.1.4; python_version >= "3"->google-auth<2,>=1.6.3->tensorboard>=1.14->pytorch_lightning==0.8.1) (0.4.8)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.14->pytorch_lightning==0.8.1) (3.1.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard>=1.14->pytorch_lightning==0.8.1) (3.4.0)
    Building wheels for collected packages: attr
      Building wheel for attr (setup.py) ... done
      Created wheel for attr: filename=attr-0.3.1-cp36-none-any.whl size=2459 sha256=5b42000be0dd70325adeeb4e1a4b5acf1c665ee199c8ae2f4d7ce718a699a23c
      Stored in directory: /root/.cache/pip/wheels/f0/96/9b/1f8892a707d17095b5a6eab0275da9d39e68e03a26aee2e726
    Successfully built attr
    Installing collected packages: attr, coremltools
    Successfully installed attr-0.3.1 coremltools-4.0

```
# 폴더 이동
%cd /content/drive/MyDrive/Colab Notebooks/gpt/KoGPT2-chatbot-master

import torch
import numpy as np
from kogpt2.pytorch_kogpt2 import get_pytorch_kogpt2_model
from gluonnlp.data import SentencepieceTokenizer
from kogpt2.utils import get_tokenizer
from pytorch_lightning.core.lightning import LightningModule
import coremltools as ct

tok_path = get_tokenizer()
model, vocab = get_pytorch_kogpt2_model()
tok = SentencepieceTokenizer(tok_path, num_best=0, alpha=0)

class KoGPT2Chat(LightningModule):
    def __init__(self, hparams, **kwargs):
        super(KoGPT2Chat, self).__init__()
        self.hparams = hparams
        self.tok_path = tok_path
        self.neg = -1e18
        self.kogpt2, self.vocab = model, vocab

        self.loss_function = torch.nn.CrossEntropyLoss(reduction='none')

    def forward(self, inputs):
        # (batch, seq_len, hiddens)
        output, _ = self.kogpt2(inputs)
        return output

# init
U_TKN = '<usr>'
S_TKN = '<sys>'
BOS = '<s>'
EOS = '</s>'
MASK = '<unused0>'
SENT = '<unused1>'

gpt_chat_model = KoGPT2Chat.load_from_checkpoint("./model_chp/model_last.ckpt")
gpt_chat_model.eval()

sent_tokens = tok("")
sent='0'
toked = tok(sent)
q_tok = tok("")
a = ''
a_tok = []

input_ids = torch.LongTensor([
                    vocab[U_TKN]] + vocab[q_tok] +
                    vocab[EOS, SENT] + vocab[sent_tokens] +
                    vocab[EOS, S_TKN] +
                    vocab[a_tok]).unsqueeze(dim=0)

# export
traced_model = torch.jit.trace(gpt_chat_model, input_ids)
traced_model.save("sample.pt")

input_shape = ct.Shape(shape=(1, ct.RangeDim(), 512))
model_input = ct.TensorType(shape=input_shape, dtype=np.int16)

# Convert the saved PyTorch model to Core ML
mlmodel = ct.convert(traced_model, inputs=[model_input])
mlmodel.save('./mlmodel.mlmodel')
print("Export of kogpt2 mlmodel complete!")
```

    WARNING:root:scikit-learn version 0.22.2.post1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
    WARNING:root:TensorFlow version 2.4.0 detected. Last version known to be fully compatible is 2.3.1 .
    WARNING:root:Keras version 2.4.3 detected. Last version known to be fully compatible of Keras is 2.2.4 .

    /content/drive/MyDrive/Colab Notebooks/gpt/KoGPT2-chatbot-master
    [██████████████████████████████████████████████████]
    [██████████████████████████████████████████████████]
    using cached model

    /usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py:146: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      w = w / (float(v.size(-1)) ** 0.5)
    /usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py:148: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      mask = self.bias[:, :, ns - nd : ns, :ns]
    Converting Frontend ==> MIL Ops: 100%|█████████▉| 1303/1306 [00:04<00:00, 301.54 ops/s]
    Running MIL optimization passes: 100%|██████████| 17/17 [00:07<00:00,  2.42 passes/s]
    Translating MIL ==> MLModel Ops: 100%|██████████| 1316/1316 [00:26<00:00, 50.06 ops/s]

    Export of kogpt2 mlmodel complete!

```

```


Uploading multiple 'documents' already processed. to be included in the main file.

Compressed pickles:
https://betterprogramming.pub/load-fast-load-big-with-compressed-pickles-5f311584507e#:~:text=Compressed%20pickle&text=It%20creates%20a%20pickle%20object,to%20the%20saved%20file%20automatically.&text=filename'%2C%20data)-,filename%20is%20the%20name%20of%20the%20file%20with%20no,data%20is%20any%20object.

```
# This is formatted as code
```

```
import bz2
import pickle
import _pickle as cPickle

# Load any compressed pickle file
def decompress_pickle(file):
 data = bz2.BZ2File(file, 'rb')
 data = cPickle.load(data)
 return data

from google.colab import files
uploaded = files.upload()
filenames = list(uploaded.keys())

```

<input type="file" id="files-0500f598-aaad-4d66-b4b8-3ff00c7b3331" name="files[]" multiple disabled
   style="border:none" />
<output id="result-0500f598-aaad-4d66-b4b8-3ff00c7b3331">
 Upload widget is only available when the cell has been executed in the
 current browser session. Please rerun this cell to enable.
 </output>

    Saving Envoy_loar.pbz2 to Envoy_loar.pbz2
    Saving TLSwiki.pbz2 to TLSwiki.pbz2

```
# initiating aggregate Conceptdata dataframe, aggregate np_to_sent, sent_to_npflat, sent_to_tags etc.
import pandas as pd
Agg_Conceptdata = pd.DataFrame()
All_Conceptdata = pd.DataFrame()
Agg_np_to_sent = dict()
Agg_sent_to_npflat = dict()
Agg_sent_to_phen = dict()
Agg_phen_to_sent = dict()
Agg_att_to_sent = dict()
Agg_sent_to_att = dict()
Agg_ins_to_sent = dict()
Agg_sent_to_ins = dict()
Agg_set_to_sent = dict()
Agg_np_to_forms = dict()
doc_to_np = dict()
np_to_doc = dict()
Agg_df = pd.DataFrame()
Agg_df = pd.DataFrame()
Agg_np_to_roles = dict()
Agg_sent_to_clt = dict()
Agg_sents = dict()
#Agg_sents_df = pd.DataFrame()
#Agg_docs_df = pd.DataFrame()
All_df = pd.DataFrame()

for filename in filenames:
  # download file with id
  #downloaded = drive.CreateFile({'id':file_to_id_dict[filename]})
  #downloaded.GetContentFile(filename)
  concepts = decompress_pickle(filename)

  Conceptdata = concepts['Conceptdata']
  sent_to_npflat = concepts['sent_to_npflat']
  np_to_sent = concepts['np_to_sent']
  np_to_forms = concepts['np_to_forms']
  sent_to_phen = concepts['sent_to_phen']
  phen_to_sent = concepts['phen_to_sent']
  sent_to_att = concepts['sent_to_att']
  att_to_sent = concepts['att_to_sent']
  att_to_sent = concepts['att_to_sent']
  ins_to_sent = concepts['ins_to_sent']
  sent_to_ins = concepts['sent_to_ins']
  set_to_sent = concepts['set_to_sent']
  sent_to_set = concepts['sent_to_set']
  np_to_roles = concepts['np_to_roles']
  sent_to_clt = concepts['sent_to_clt']
  sents = concepts['sents']
  df = concepts['df']

  Conceptdata['docname'] = filename
  Agg_Conceptdata = Agg_Conceptdata.append(Conceptdata,ignore_index=True)

  Agg_sent_to_clt[filename.replace(".pbz2","")] = sent_to_clt
  Agg_np_to_sent[filename.replace(".pbz2","")] = np_to_sent
  Agg_sents[filename.replace(".pbz2","")] = sents
  Agg_sent_to_npflat[filename.replace(".pbz2","")] = sent_to_npflat

  Agg_df = Agg_df.append(df,ignore_index=True)
  doc_to_np[filename] = list(np_to_sent.keys())

  for np in np_to_sent:
    # if np in Agg_np_to_sent:
    #   Agg_np_to_sent[np] = Agg_np_to_sent[np] + [(filename,s) for s in np_to_sent[np]]
    # else:
    #   Agg_np_to_sent[np] = [(filename,s) for s in np_to_sent[np]]

    if np in np_to_doc:
      np_to_doc[np] = np_to_doc[np] + [filename]
    else:
      np_to_doc[np] = [filename]

  for np in np_to_forms:
    if np in Agg_np_to_forms:
      Agg_np_to_forms[np] = Agg_np_to_forms[np] + np_to_forms[np]
    else:
      Agg_np_to_forms[np] = np_to_forms[np]

  for np in np_to_roles:
    if np in Agg_np_to_roles:
      Agg_np_to_roles[np] = Agg_np_to_roles[np] + np_to_roles[np]
    else:
      Agg_np_to_roles[np] = np_to_roles[np]

  for np in phen_to_sent:
    if np in Agg_phen_to_sent:
      Agg_phen_to_sent[np] = Agg_phen_to_sent[np] + [(filename,s) for s in phen_to_sent[np]]
    else:
      Agg_phen_to_sent[np] = [(filename,s) for s in phen_to_sent[np]]

  for np in att_to_sent:
    if np in Agg_att_to_sent:
      Agg_att_to_sent[np] = Agg_att_to_sent[np] + [(filename,s) for s in att_to_sent[np]]
    else:
      Agg_att_to_sent[np] =  [(filename,s) for s in att_to_sent[np]]

  for np in set_to_sent:
    if np in Agg_set_to_sent:
      Agg_set_to_sent[np] = Agg_set_to_sent[np] + [(filename,s) for s in set_to_sent[np]]
    else:
      Agg_set_to_sent[np] =  [(filename,s) for s in set_to_sent[np]]

  for np in ins_to_sent:
    if np in Agg_ins_to_sent:
      Agg_ins_to_sent[np] = Agg_ins_to_sent[np] + [(filename,s) for s in ins_to_sent[np]]
    else:
      Agg_ins_to_sent[np] =  [(filename,s) for s in ins_to_sent[np]]

```

### Create concept inventory dataframe with some metrics:
- Concept: concept phrase
- Occurence: list of sentences in which the phrase occurs
- Frequency: number of sentences in which the phrase occurs
- Mean: average of sentence numbers in the text in which the phrase occurs normalized to number of sentences
- Median: median of sentence numbers in the text in which the phrase occurs normalized to number of sentences. Lets us know if phrase occurs much more in the beginning of the text, or towards the end. can indicate how central the phrase is to the text.
- Sdev: standard deviation of the sentences in which the phrase occurs (indicates the dispersion of the phrase in the text, is it important all throughout, or specific to a sub-topic? The higher the value, the more dispersed)

```
import math
import numpy as num
All_Conceptdata = pd.DataFrame()

def most_common_form(np):
  return pd.Series(Agg_np_to_forms[np]).value_counts().sort_values(ascending=False).index[0]

Agg_np_to_mcform = dict()
for np in Agg_np_to_forms:
  Agg_np_to_mcform[np] = most_common_form(np)

All_Conceptdata = Agg_Conceptdata.groupby('Concept').agg(doc_Occurence = pd.NamedAgg('docname',lambda x: list(x)),
                                                          doc_Frequency = pd.NamedAgg('docname',lambda x: x.shape[0]),
                                                          Raw_Frequency = pd.NamedAgg('Frequency','sum'),
                                                          Mean = pd.NamedAgg('Mean','mean'),
                                                          Median = pd.NamedAgg('Median','mean'),
                                                          Sdev = pd.NamedAgg('Sdev','mean'),
                                                          Ext_IDF = pd.NamedAgg('IDF',num.nanmin))

All_Conceptdata['Mean_Frequency'] =  All_Conceptdata['Raw_Frequency']/All_Conceptdata['doc_Frequency']
All_Conceptdata['normalized_RawFreq'] = All_Conceptdata['Raw_Frequency']/All_Conceptdata['Raw_Frequency'].max()
All_Conceptdata['normalized_MeanFreq'] = All_Conceptdata['Mean_Frequency']/All_Conceptdata['Mean_Frequency'].max()
All_Conceptdata['intIDF'] = All_Conceptdata['doc_Frequency'].apply(lambda x: math.log(len(filenames),2)-abs(math.log(1+x,2)))
All_Conceptdata['intmeanTFIDF'] = All_Conceptdata['normalized_MeanFreq']*All_Conceptdata['intIDF']
for filename in filenames:
  colname = filename.replace(".pbz2","")
  All_Conceptdata = pd.merge(left = All_Conceptdata,
                             right = Agg_Conceptdata.loc[Agg_Conceptdata['docname']==filename,['Concept','Frequency']],
                             how='left',
                             left_on = 'Concept',
                             right_on = 'Concept')
  All_Conceptdata[colname+'_TF'] = All_Conceptdata['Frequency']
  del All_Conceptdata['Frequency']
  All_Conceptdata[colname+'_TF'].fillna(0,inplace=True)
  All_Conceptdata[colname+'_IntTFIDF'] = All_Conceptdata[colname+'_TF']*All_Conceptdata['intIDF']

All_Conceptdata['MCForm'] = All_Conceptdata['Concept'].apply(lambda x: Agg_np_to_mcform[x])

All_Conceptdata['role_frac'] = All_Conceptdata['Concept'].apply(lambda x: dict(pd.Series(Agg_np_to_roles[x]).value_counts(normalize=True)))
All_Conceptdata['phen_frac'] = All_Conceptdata['role_frac'].apply(lambda x: x.get('phen',0))
All_Conceptdata['att_frac'] = All_Conceptdata['role_frac'].apply(lambda x: x.get('att',0))
All_Conceptdata['set_frac'] = All_Conceptdata['role_frac'].apply(lambda x: x.get('set',0))
All_Conceptdata['ins_frac'] = All_Conceptdata['role_frac'].apply(lambda x: x.get('ins',0))
del All_Conceptdata['role_frac']

#All_Conceptdata['phen_frac'] = All_Conceptdata
```

```
All_df = pd.DataFrame()
Agg_df['tuple'] = Agg_df[['Concept1','Concept2']].apply(lambda x:tuple(x),axis=1)
All_df = Agg_df.groupby('tuple').agg(Concept1 = pd.NamedAgg('Concept1',lambda x: list(x)[0]),
                            Concept2 = pd.NamedAgg('Concept2',lambda x: list(x)[0]),
                            Bondstrength = pd.NamedAgg('Bondstrength','sum'),
                            mean_dAB = pd.NamedAgg('dAB',num.nanmean),
                            mean_dBA = pd.NamedAgg('dBA',num.nanmean),
                            ExtIDFA = pd.NamedAgg('IDFA',num.nanmean),
                            ExtIDFB = pd.NamedAgg('IDFB',num.nanmean),
                            SdevA = pd.NamedAgg('SdevA',num.nanmean),
                            SdevB = pd.NamedAgg('SdevB',num.nanmean),

                            )

All_df = pd.merge(left = All_df,right = All_Conceptdata.loc[:,['Concept','Raw_Frequency']],how="left",left_on = 'Concept1',right_on='Concept')
All_df['Raw_TFA'] = All_df['Raw_Frequency']
All_df['Raw_TFA'].fillna(0)
del All_df['Raw_Frequency']
All_df['Ext_TFIDFA'] = All_df['Raw_TFA']*All_df['ExtIDFA']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','Raw_Frequency']],how="left",left_on = 'Concept2',right_on='Concept')
All_df['Raw_TFB'] = All_df['Raw_Frequency']
All_df['Raw_TFB'].fillna(0)
del All_df['Raw_Frequency']
All_df['Ext_TFIDFB'] = All_df['Raw_TFB']*All_df['ExtIDFB']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','MCForm']],how="left",left_on = 'Concept1',right_on='Concept')
All_df['MCForm1'] = All_df['MCForm']
All_df['MCForm1'].fillna('')
del All_df['MCForm']
 #All_df['Concept_x']#, All_df['Concept_y']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','MCForm']],how="left",left_on = 'Concept2',right_on='Concept')
All_df['MCForm2'] = All_df['MCForm']
All_df['MCForm2'].fillna('')
del All_df['MCForm']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','phen_frac']],how="left",left_on = 'Concept1',right_on='Concept')
All_df['phen_fracA'] = All_df['phen_frac']
All_df['phen_fracA'].fillna(0)
del All_df['phen_frac']
 #All_df['Concept_x']#, All_df['Concept_y']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','phen_frac']],how="left",left_on = 'Concept2',right_on='Concept')
All_df['phen_fracB'] = All_df['phen_frac']
All_df['phen_fracB'].fillna(0)
del All_df['phen_frac']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','att_frac']],how="left",left_on = 'Concept1',right_on='Concept')
All_df['att_fracA'] = All_df['att_frac']
All_df['att_fracA'].fillna(0)
del All_df['att_frac']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','att_frac']],how="left",left_on = 'Concept2',right_on='Concept')
All_df['att_fracB'] = All_df['att_frac']
All_df['att_fracB'].fillna(0)
del All_df['att_frac']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','set_frac']],how="left",left_on = 'Concept1',right_on='Concept')
All_df['set_fracA'] = All_df['set_frac']
All_df['set_fracA'].fillna(0)
del All_df['set_frac']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','set_frac']],how="left",left_on = 'Concept2',right_on='Concept')
All_df['set_fracB'] = All_df['set_frac']
All_df['set_fracB'].fillna(0)
del All_df['set_frac']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','ins_frac']],how="left",left_on = 'Concept1',right_on='Concept')
All_df['ins_fracA'] = All_df['ins_frac']
All_df['ins_fracA'].fillna(0)
del All_df['ins_frac']

All_df = pd.merge(left= All_df,right = All_Conceptdata.loc[:,['Concept','ins_frac']],how="left",left_on = 'Concept2',right_on='Concept')
All_df['ins_fracB'] = All_df['ins_frac']
All_df['ins_fracB'].fillna(0)
del All_df['ins_frac']

for filename in filenames:
  colname = filename.replace(".pbz2","")
  All_df = pd.merge(left = All_df,
                             right = All_Conceptdata.loc[:,['Concept',colname+'_IntTFIDF']],
                             how='left',
                             left_on = 'Concept1',
                             right_on = 'Concept')
  All_df[colname+'_IntTFIDFA'] = All_df[colname+'_IntTFIDF']
  del All_df[colname+'_IntTFIDF']
  All_df[colname+'_IntTFIDFA'].fillna(0,inplace=True)
  #del All_df['Concept_x'], All_df['Concept_y']

  All_df = pd.merge(left = All_df,
                             right = All_Conceptdata.loc[:,['Concept',colname+'_IntTFIDF']],
                             how='left',
                             left_on = 'Concept2',
                             right_on = 'Concept')
  All_df[colname+'_IntTFIDFB'] = All_df[colname+'_IntTFIDF']
  del All_df[colname+'_IntTFIDF']
  All_df[colname+'_IntTFIDFB'].fillna(0,inplace=True)

try:
  del All_df['Concept_x']
except:
  pass

try:
  del All_df['Concept_y']
except:
  pass

import networkx as nx
allG = nx.from_pandas_edgelist(All_df,'Concept1','Concept2')
centrality_dict = nx.eigenvector_centrality_numpy(allG)
centrality = pd.Series(centrality_dict)
centrality.name = 'Centrality'
centrality = pd.DataFrame(centrality)

centrality['Concept'] = list(centrality.index)
All_Conceptdata = pd.merge(left=All_Conceptdata,right=centrality[['Concept','Centrality']],how = 'left',left_on = 'Concept',right_on = 'Concept',suffixes=("", ""))

# centrality

df['Centrality_A'] = df['Concept1'].apply(lambda x:
                                          list(Conceptdata.loc[Conceptdata['Concept']==x,'Centrality'].values)[0])

df['Centrality_B'] = df['Concept2'].apply(lambda x:
                                          list(Conceptdata.loc[Conceptdata['Concept']==x,'Centrality'].values)[0])

nonos = ['openstax','openstax book','http','website','credit','wikipedia','book','chapter','wikibook','wikibook project']
df1 = All_df.loc[~All_df['Concept1'].isin(nonos)]
df1 = df1.loc[~df1['Concept2'].isin(nonos)]
All_df = df1.copy()
del df1
```

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:34: FutureWarning: Passing 'suffixes' which cause duplicate columns {'Concept_x'} in the result is deprecated and will raise a MergeError in a future version.
    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:45: FutureWarning: Passing 'suffixes' which cause duplicate columns {'Concept_x'} in the result is deprecated and will raise a MergeError in a future version.
    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:56: FutureWarning: Passing 'suffixes' which cause duplicate columns {'Concept_x'} in the result is deprecated and will raise a MergeError in a future version.
    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:66: FutureWarning: Passing 'suffixes' which cause duplicate columns {'Concept_x'} in the result is deprecated and will raise a MergeError in a future version.
    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:76: FutureWarning: Passing 'suffixes' which cause duplicate columns {'Concept_x'} in the result is deprecated and will raise a MergeError in a future version.
    /usr/local/lib/python3.7/dist-packages/pandas/core/generic.py:6392: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      return self._update_inplace(result)
    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:98: FutureWarning: Passing 'suffixes' which cause duplicate columns {'Concept_x'} in the result is deprecated and will raise a MergeError in a future version.

```
All_Conceptdata.columns
```

    Index(['Concept', 'doc_Occurence', 'doc_Frequency', 'Raw_Frequency', 'Mean',
           'Median', 'Sdev', 'Ext_IDF', 'Mean_Frequency', 'normalized_RawFreq',
           'normalized_MeanFreq', 'intIDF', 'intmeanTFIDF', 'Envoy_loar_TF',
           'Envoy_loar_IntTFIDF', 'TLSwiki_TF', 'TLSwiki_IntTFIDF', 'MCForm',
           'phen_frac', 'att_frac', 'set_frac', 'ins_frac', 'Centrality'],
          dtype='object')

```
!pip install dimcli networkx pyvis -U --quiet
from dimcli.utils.networkviz import *
import networkx as nx
```

         |████████████████████████████████| 214 kB 7.7 MB/s
         |████████████████████████████████| 51 kB 6.4 MB/s

```
def filter_by_concept(df,concept_x,number = 3):
  dfsub1 = df.loc[df['Concept1']==concept_x,['Concept1','Concept2','MCForm1','MCForm2','Raw_TFB','mean_dAB','Ext_TFIDFA','Ext_TFIDFB','SdevB',
                                             'phen_fracA','phen_fracB','set_fracA','set_fracB','ins_fracA','ins_fracB','att_fracA','att_fracB']].copy()

  dfsub1['Concept_y'] = dfsub1['Concept2']
  dfsub1['MCForm_y'] = dfsub1['MCForm2']
  dfsub1['Cooccurence_cc_y'] = dfsub1['mean_dAB']
  dfsub1['F_y'] = dfsub1['Raw_TFB']
  dfsub1['TFIDF_y'] = dfsub1['Ext_TFIDFB']
  dfsub1['Sdev_y'] = dfsub1['SdevB']
  dfsub1['Concept_x'] = dfsub1['Concept1']
  dfsub1['MCForm_x'] = dfsub1['MCForm1']
  dfsub1['phen_frac_x'] = dfsub1['phen_fracA']
  dfsub1['phen_frac_y'] = dfsub1['phen_fracB']
  dfsub1['att_frac_x'] = dfsub1['att_fracA']
  dfsub1['att_frac_y'] = dfsub1['att_fracB']
  dfsub1['set_frac_x'] = dfsub1['set_fracA']
  dfsub1['set_frac_y'] = dfsub1['set_fracB']
  dfsub1['ins_frac_x'] = dfsub1['ins_fracA']
  dfsub1['ins_frac_y'] = dfsub1['ins_fracB']

  dfsub2 = df.loc[df['Concept2']==concept_x,['Concept1','Concept2','MCForm1','MCForm2','Raw_TFA','mean_dBA','Ext_TFIDFA','Ext_TFIDFB','SdevA',
                                             'phen_fracA','phen_fracB','set_fracA','set_fracB','ins_fracA','ins_fracB','att_fracA','att_fracB']].copy()
  dfsub2['Sdev_y'] = dfsub2['SdevA']
  dfsub2['Concept_y'] = dfsub2['Concept1']
  dfsub2['MCForm_y'] = dfsub2['MCForm1']
  dfsub2['Cooccurence_cc_y'] = dfsub2['mean_dBA']
  dfsub2['F_y'] = dfsub2['Raw_TFA']
  dfsub2['TFIDF_y'] = dfsub2['Ext_TFIDFA']
  dfsub2['Concept_x'] = dfsub2['Concept2']
  dfsub2['MCForm_x'] = dfsub2['MCForm2']

  dfsub2['phen_frac_x'] = dfsub2['phen_fracB']
  dfsub2['phen_frac_y'] = dfsub2['phen_fracA']
  dfsub2['att_frac_x'] = dfsub2['att_fracB']
  dfsub2['att_frac_y'] = dfsub2['att_fracA']
  dfsub2['set_frac_x'] = dfsub2['set_fracB']
  dfsub2['set_frac_y'] = dfsub2['set_fracA']
  dfsub2['ins_frac_x'] = dfsub2['ins_fracB']
  dfsub2['ins_frac_y'] = dfsub2['ins_fracA']

  dfsub = dfsub1.append(dfsub2)
  if dfsub1.shape[0]>0:
    TFIDF_cutoff = dfsub1['Ext_TFIDFA'].head(1).item()
  else:
    TFIDF_cutoff = dfsub2['Ext_TFIDFB'].head(1).item()
  dfsub = dfsub.loc[:,['Concept_x','Concept_y','MCForm_x','MCForm_y','F_y','TFIDF_y','Sdev_y','Cooccurence_cc_y',
                       'phen_frac_x','phen_frac_y','att_frac_x','att_frac_y','set_frac_x','set_frac_y','ins_frac_x','ins_frac_y']]
  dfsub = dfsub.drop_duplicates()
  dfsub['TFIDFbydAB'] = dfsub['TFIDF_y']/dfsub['Cooccurence_cc_y']
  dfsub = dfsub.loc[dfsub['TFIDF_y']<TFIDF_cutoff,:].sort_values(by=['TFIDFbydAB'],ascending=[False]).head(number)
  y_conceptsset = set(dfsub['Concept_y'])
  return (dfsub,y_conceptsset)

def filter_by_listofconcepts(df,listofconcepts,number = 3):
  dfsublist = pd.DataFrame()
  yconceptsetlist = set()
  for c in listofconcepts:
    dfsubeach, y_conceptset = filter_by_concept(df,c,number)
    dfsublist = dfsublist.append(dfsubeach)
    yconceptsetlist = yconceptsetlist.union(y_conceptset)
  dfsublist = dfsublist.drop_duplicates()
  return dfsublist,yconceptsetlist
```

```
# def filter_by_concept(df,concept_x,number = 3):
#   dfsub1 = df.loc[df['Concept1']==concept_x,['Concept1','Concept2','MCForm1','MCForm2','Raw_TFB','mean_dAB','Ext_TFIDFA','Ext_TFIDFB']].copy()
#   dfsub1['Concept_y'] = dfsub1['Concept2']
#   dfsub1['MCForm_y'] = dfsub1['MCForm2']

#   dfsub1['Cooccurence_cc_y'] = dfsub1['mean_dAB']
#   dfsub1['F_y'] = dfsub1['Raw_TFB']
#   dfsub1['TFIDF_y'] = dfsub1['Ext_TFIDFB']
#   dfsub1['Concept_x'] = dfsub1['Concept1']
#   dfsub1['MCForm_x'] = dfsub1['MCForm1']
#   dfsub2 = df.loc[df['Concept2']==concept_x,['Concept1','Concept2','MCForm1','MCForm2','Raw_TFA','mean_dBA','Ext_TFIDFA','Ext_TFIDFB']].copy()
#   dfsub2['Concept_y'] = dfsub2['Concept1']
#   dfsub2['MCForm_y'] = dfsub2['MCForm1']

#   dfsub2['Cooccurence_cc_y'] = dfsub2['mean_dBA']
#   dfsub2['F_y'] = dfsub2['Raw_TFA']
#   dfsub2['TFIDF_y'] = dfsub2['Ext_TFIDFA']
#   dfsub2['Concept_x'] = dfsub2['Concept2']
#   dfsub2['MCForm_x'] = dfsub2['MCForm2']
#   dfsub = dfsub1.append(dfsub2)
#   if dfsub1.shape[0]>0:
#     TFIDF_cutoff = dfsub1['Ext_TFIDFA'].head(1).item()
#   else:
#     TFIDF_cutoff = dfsub2['Ext_TFIDFB'].head(1).item()
#   dfsub = dfsub.loc[:,['Concept_x','Concept_y','MCForm_x','MCForm_y','Cooccurence_cc_y','TFIDF_y']].copy()
#   dfsub = dfsub.drop_duplicates()
#   dfsub['TFIDFbydAB'] = dfsub['TFIDF_y']/dfsub['Cooccurence_cc_y']
#   dfsub = dfsub.loc[dfsub['TFIDF_y']<TFIDF_cutoff,:].sort_values(by=['TFIDFbydAB'],ascending=[False]).head(number)
#   y_conceptsset = set(dfsub['Concept_y'])
#   return (dfsub,y_conceptsset)

```

```
# def filter_by_listofconcepts(df,listofconcepts,number = 3):
#   dfsublist = pd.DataFrame()
#   yconceptsetlist = set()
#   for c in listofconcepts:
#     dfsubeach, y_conceptset = filter_by_concept(df,c,number)
#     dfsublist = dfsublist.append(dfsubeach)
#     yconceptsetlist = yconceptsetlist.union(y_conceptset)
#   dfsublist = dfsublist.drop_duplicates()
#   return dfsublist,yconceptsetlist
```

```
#@title Document centric view: Most important concepts in documents { run: "auto", vertical-output: true }
import numpy as num
net = NetworkViz(height='700px', width='700px', directed=False,notebook=True)
included_docs = ["Envoy_loar"] #@param
Centrality_Detail = 100 #@param {type:"slider", min:1, max:100, step:1}
Frequency_Detail = 100 #@param {type:"slider", min:1, max:100, step:1}
IDF_Detail = 100#@param {type:"slider", min:1, max:100, step:1}
maxnumber = 34#@param {type:"slider", min:1, max:50, step:1}
Frequency_cutoff = num.quantile(All_Conceptdata['Raw_Frequency'],(100-Frequency_Detail)/100)
IDF_cutoff = num.nanquantile(All_Conceptdata['Ext_IDF'],(100-IDF_Detail)/100)
Centrality_cutoff = num.nanquantile(All_Conceptdata['Centrality'],(100-Centrality_Detail)/100)
#print(FA,FB)
docnodes = []
if included_docs!=["All"]:

  for d in included_docs:
    docnodes = docnodes + doc_to_np[d+'.pbz2']
  docnodes = set(docnodes)
  included_nodes = list(All_Conceptdata.loc[(All_Conceptdata['Concept'].isin(docnodes)) & (All_Conceptdata['Centrality']>Centrality_cutoff) & (All_Conceptdata['Raw_Frequency']>Frequency_cutoff) & (All_Conceptdata['Ext_IDF']>IDF_cutoff),'Concept'])
else:
  for d in filenames:
    docnodes = docnodes + doc_to_np[d]
  included_nodes = list(All_Conceptdata.loc[ (All_Conceptdata['Centrality']>Centrality_cutoff) & (All_Conceptdata['Raw_Frequency']>Frequency_cutoff) & (All_Conceptdata['Ext_IDF']>IDF_cutoff),'Concept'])
  #docnodes =
dfnoones = All_df.loc[
                  (All_df['Concept1'].isin(included_nodes)) &
                  (All_df['Concept2'].isin(included_nodes))
                  #(df['FA']>Frequency_cutoff) &
                  #(df['IDFA']>IDF_cutoff) &
                  #(df['FB']>Frequency_cutoff) &
                  #(df['IDFB']>IDF_cutoff)
                  ,:].head(maxnumber)
print('Number of edges: ',dfnoones.shape[0])
print('Percentage of all concepts present in selected documents',len(docnodes)/All_Conceptdata.shape[0]*100)
print('Most central concepts left out:')
print(All_Conceptdata.loc[~All_Conceptdata['Concept'].isin(docnodes),:].sort_values(by='Centrality',ascending=False).head(20)[['Concept','Raw_Frequency']])
#print(dfnoones)
#dfnoones['size'] = dfnoones['FA']
bigG = nx.from_pandas_edgelist(dfnoones,'MCForm1','MCForm2')
# getting the nodes in bigG
nodes = bigG.nodes
# size_series = Conceptdata.loc[Conceptdata['Concept'].isin(nodes),['Concept','Frequency']].set_index('Concept')
# size_dict = size_series.to_dict()['Frequency']
size_series = Conceptdata.loc[Conceptdata['Concept'].isin(nodes),['Concept','Centrality']].set_index('Concept')
size_dict = size_series.to_dict()['Centrality']
#tiered sizes
numbins = int(maxnumber/5)
#sizelabels = num.linspace(start = 35,stop = 5,num=numbins)
#print(numbins,sizelabels)
size_series['binned'] = pd.qcut(size_series['Centrality'],numbins,duplicates='drop')
unique_bins = size_series['binned'].unique()
unique_bins_sorted = list(pd.Series(unique_bins).sort_values(ascending=False))
print(unique_bins)
if len(unique_bins)>2:
  sizelabels = num.linspace(start = 45,stop = 5, num = len(unique_bins))
  size_series['labeled'] = size_series['binned'].apply(lambda x: dict(zip(unique_bins_sorted,sizelabels))[x])
else:
  size_series['labeled'] = 45
nx.set_node_attributes(bigG,size_series['labeled'],name='size')
#size_series['size']
#nx.set_node_attributes(bigG
#net('500px','500px')
net.from_nx(bigG)
#net.show_buttons()
net.show('nx.html')
```

    Number of edges:  34
    Percentage of all concepts present in selected documents 34.55223880597015
    Most central concepts left out:
                             Concept  Raw_Frequency
    81                        attack             31
    83                      attacker             20
    1090                         ssl             43
    1277                        user             16
    1298               vulnerability             17
    125                      browser              7
    7                - middle attack             10
    449              forward secrecy             15
    1212                     traffic              7
    1037                     session             13
    674   man - in - - middle attack              9
    1289                     version             19
    987                      secrecy              8
    850                  private key              6
    363                   encryption             14
    1312                  web server              8
    1274                         use             11
    1187           tl implementation              3
    599                 key exchange              7
    495                      hellman              8
    [(0.0615, 0.0713], (0.0477, 0.0615], (0.0376, 0.041], (0.031, 0.0376], (0.041, 0.0477], (0.023799999999999998, 0.031]]
    Categories (6, interval[float64, right]): [(0.023799999999999998, 0.031] < (0.031, 0.0376] < (0.0376, 0.041] <
                                               (0.041, 0.0477] < (0.0477, 0.0615] < (0.0615, 0.0713]]

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.16.1/vis.css" type="text/css" />

```
#@title Concept centric view: Hierarchical radial branching visualization { run: "auto", vertical-output: true }
from dimcli.utils.networkviz import *
#from dimcli.core.extras import NetworkViz
net = NetworkViz(height='500px', width='500px', directed=False,notebook=True)
listnodes = list(nodes)
central_concept = "alpn" #@param ""
#TFIDF_cutoff = 0.95 #@param {type:"slider", min:0, max:1, step:0.05}
#@markdown Radius (max path length from central concept)
radius = 2 #@param {type:"slider", min:0, max:5, step:1}
#Cooccurence_cutoff = 0.2
#@markdown Number of splits
splits = 5 #@param {type:"slider", min:0, max:5, step:1}

radiuscolors = ["#f50c0c","#ff9900",'#e5ff00','#00ff1e',"#00ffe5"]
color_dict = dict()
color_dict[central_concept] = radiuscolors[0]
dfsub = pd.DataFrame()
y_conceptsnew = set([central_concept])
y_concepts = set()
y_concepts = y_concepts.union(y_conceptsnew)
for i in range(radius):
  dfsubnew, y_conceptsnew = filter_by_listofconcepts(All_df,y_conceptsnew,splits)
  y_concepts = y_concepts.union(y_conceptsnew)
  for yc in y_conceptsnew:
    if yc not in color_dict:
      color_dict[yc] = radiuscolors[i]
  print(len(y_concepts))

(dfsub,yconceptsnew) = filter_by_listofconcepts(All_df,y_concepts)
for yc in y_conceptsnew:
  if yc not in color_dict:
    color_dict[yc] = radiuscolors[-1]
dfsub = dfsub.drop_duplicates()
degree_graph = dfsub.copy()

# degree graph should return the MCForms too.

#degree_graph['central_concept'] = [central_concept]*degree_graph.shape[0]
#TFIDF_threshold = num.quantile(dfsub['TFIDF_y'],TFIDF_cutoff)
#Cooccurence_threshold = num.quantile(dfsub['TFIDF_y'],Cooccurence_cutoff)
#degree_graph = degree_graph.loc[(degree_graph['TFIDF_y']>TFIDF_threshold) & (degree_graph['Cooccurence_cc_y']<Cooccurence_threshold),:].copy()
print(len(degree_graph))

G = nx.from_pandas_edgelist(degree_graph,'MCForm_x','MCForm_y', create_using=nx.Graph())
nx.set_node_attributes(G,color_dict,"color")
nx.set_node_attributes(G,{central_concept:20},"size")
#G.set_node_attributes()

net.from_nx(G)
net.show('nx.html')

# dfnoones = df.loc[(df['FA']>7) & (df['FB']>5),:]
# bigG = nx.from_pandas_edgelist(dfnoones,'Concept1','Concept2')
# #net('500px','500px')
# net.from_nx(bigG)
# #net.show_buttons()
# net.show('nx.html')

```

    6
    24
    68

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.16.1/vis.css" type="text/css" />

```
All_Conceptdata.loc[All_Conceptdata['MCForm']=='TLS protocol']
```

    Warning: Total number of columns (23) exceeds max_columns (20). Falling back to pandas display.

      Concept
      doc_Occurence
      doc_Frequency
      Raw_Frequency
      Mean
      Median
      Sdev
      Ext_IDF
      Mean_Frequency
      normalized_RawFreq
      normalized_MeanFreq
      intIDF
      intmeanTFIDF
      Envoy_loar_TF
      Envoy_loar_IntTFIDF
      TLSwiki_TF
      TLSwiki_IntTFIDF
      MCForm
      phen_frac
      att_frac
      set_frac
      ins_frac
      Centrality

      1196
      tl protocol
      [TLSwiki.pbz2]
      1
      4
      0.314309
      0.229904
      0.325953
      9.146468
      4.0
      0.035088
      0.070175
      0.0
      0.0
      0.0
      0.0
      4.0
      0.0
      TLS protocol
      0.0
      0.0
      1.0
      0.0
      0.027477

      <button class="colab-df-convert" onclick="convertToInteractive('df-f8057696-ab1e-44ba-adb3-9f67a3cc3047')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

```
# dfsub, yconcept_list = filter_by_listofconcepts(All_df,listofconcepts = ['teacher'],number = 250)
# dfsub.loc[dfsub['Concept_y'].isin(list(phen_to_sent.keys())),:].sort_values(by='TFIDFbydAB',ascending=False)

dfsub, yconcept_list = filter_by_listofconcepts(All_df,listofconcepts = ['learning'],number = 1250)
dfsub.loc[dfsub['Concept_y'].isin(list(phen_to_sent.keys())),:].sort_values(by=['F_y','phen_frac_y','TFIDFbydAB'],ascending=[False]*3)
```

```
#@title Visualizing Relationships/Path between two concepts { run: "auto", vertical-output: true }
net1 = NetworkViz(height='500px', width='500px', directed=False,notebook=True)
concept1 = "envoy listener"  #@param listnodes
concept2 = "circuit breaker"  #@param listnodes
cutoff = 12 #@param {type:"slider", min:8, max:20, step:1}
maxnumpaths = 10 #@param {type:"slider", min:1, max:20, step:1}
concept1 = Agg_np_to_mcform[concept1]
concept2 = Agg_np_to_mcform[concept2]
bigG = nx.from_pandas_edgelist(All_df,'MCForm1','MCForm2',create_using = nx.Graph())

from itertools import islice
def k_shortest_paths(G, source, target, k, weight=None):
    return list(
        islice(nx.shortest_simple_paths(G, source, target, weight=weight), k)
    )

def k_paths(G, source, target, k, cutoff=10):
    return list(
        islice(nx.all_simple_paths(G, source, target,cutoff=cutoff), k)
    )

if bigG.has_edge(concept1,concept2) or bigG.has_edge(concept2,concept1):
  #cutoff = 4
  spconceptset =set([concept1,concept2])
  subG = nx.subgraph(bigG,list(spconceptset))
elif nx.has_path(bigG,concept1,concept2) is False:
  print("No path found between ",concept1,"and",concept2,"!")
else:
  #print('has path')
  spconceptset = set()
  shortestpaths = nx.all_shortest_paths(bigG, concept1, concept2, method='dijkstra')
  # for sp in shortestpaths:
  #   #print(sp)
  #   spconceptset = spconceptset.union(set(sp))
  #   cutoff = len(sp)+3
  conceptset = set([concept1,concept2])
  for path in k_shortest_paths(bigG, concept1, concept2, maxnumpaths):
    #print(path)
    conceptset = conceptset.union(set(path))
  #paths = nx.all_simple_paths(bigG, concept1, concept2, cutoff=cutoff)
  subG = nx.subgraph(bigG,list(conceptset))

nx.set_node_attributes(subG,{concept1:25,concept2:25},"size")
nx.set_node_attributes(subG,{concept1:"#ff9900",concept2:"#ff9900"},"color")
net1.from_nx(subG)
net1.show('nx.html')

```

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.16.1/vis.css" type="text/css" />

```
#@title Functions for calculating memory load

def plot_clt(sent_to_clt):

    from matplotlib import pyplot as plt
    plt.xlabel('document sentence #')
    plt.ylabel('Load added to working memory by sentence')
    plt.title('Cognitive Load for '+filename)
    plt.plot(list(range(1,len(sent_to_npflat)+1)),sent_to_clt,drawstyle='steps')
    #plt.savefig('cltfig1.png')
    return None

#sent_to_clt = calc_cl_per_sentence(Agg_sent_to_npflat)
#plot_clt(sent_to_clt)

def print_sents_by_target_cl(target,cl):
    return [sents[s] for s in Agg_np_to_sent[target] if Agg_sent_to_clt[s]==cl]

```

```
#@title Functions to get blurbs of text
from sympy import Interval, Union, FiniteSet, solveset
import pandas as pd

def calc_cl_per_sentence(doc,known_concepts = [], max_idf=9.1):
    sent_to_clt = []
    sent_to_npflat = Agg_sent_to_npflat[doc]
    for i in range(len(sent_to_npflat)):
        npinsent = sent_to_npflat[i]
        npinsent = [np for np in npinsent if np not in known_concepts]
        cltinsent = [All_Conceptdata.loc[All_Conceptdata['Concept']==np,'Ext_IDF'].values for np in npinsent]
        clt= 0
        for (np,idf) in zip(npinsent,cltinsent):
            if (idf>=max_idf):
                clt = clt + 1
        sent_to_clt.append(clt)
    return sent_to_clt

def calc_clt_blurb_order(tuplist,doc,known_concepts,max_idf):
    sent_to_clt = calc_cl_per_sentence(doc,known_concepts,max_idf)
    tup_to_clt = {}
    for tup in tuplist:
        blurb_clt = 0
        for i in range(tup[0],tup[1]+1):
            blurb_clt = blurb_clt + sent_to_clt[i]
        tup_to_clt[tup] = blurb_clt
    tup_to_clt = pd.Series(tup_to_clt)
    tup_to_clt.sort_values(ascending=True)
    return tup_to_clt.sort_values(ascending=True)

def get_sentence_indices(npset,doc,max_distance=3):
    #sentslist = [np_to_sent[np1] for np1 in npset]
    # get pairs?
    np_to_sent = Agg_np_to_sent[doc]

    if len(npset)==1:
      tuplist = [(i[1],i[1]+1) for i in np_to_sent[list(npset)[0]]]
    else:
      pairs = set([frozenset([np1,np2]) for np1 in npset for np2 in npset if np1!=np2])
      #print(pairs)
      tuplist = []
      for p in pairs:
        p = list(p)
        np1 = p[0]
        np2 = p[1]

        sents1 = np_to_sent.get(np1,[])
        sents2 = np_to_sent.get(np2,[])
        print(sents1,sents2)
        ind1 = 0
        ind2 = 0

        lensents1 = len(sents1)
        #print(lensents1)
        lensents2 = len(sents2)
        #print(lensents2)
        while(ind1<lensents1 and ind2 <lensents2):
            #print(ind1,ind2)
            if (sents1[ind1]<sents2[ind2]):
                #print('sent1 less than sent2')
                if sents2[ind2]-sents1[ind1]<=max_distance:
                    tuplist.append((sents1[ind1],sents2[ind2]))
                    ind1 = ind1+1
                    ind2 = ind2 + 1
                else:
                    #ind1 = bs.bisect_left(sents1,sents2[ind2])
                    ind1 = ind1 + 1
            elif (sents1[ind1]>sents2[ind2]):
                #print('sent2 less than sent1')
                if sents1[ind1]-sents2[ind2] <= max_distance:
                    tuplist.append((sents2[ind2],sents1[ind1]))
                    ind1 = ind1 + 1
                    ind2 = ind2 + 1
                else:
                    #ind2 = bs.bisect_left(sents2,sents1[ind1])
                    ind2 = ind2 + 1
            else:
                tuplist.append((sents1[ind1],sents2[ind2]))
                ind1 = ind1+1
                ind2 = ind2+1
    #print(tuplist)
    if len(tuplist)>1:
      interval_list = [Interval(tup[0],tup[1]+1,False,False) for tup in tuplist]
      interval_union = interval_list[0]

      for i in interval_list[1:]:
        interval_union = Union(interval_union,i)
      #print(interval_list,interval_union)
      #print(interval_union.boundary,interval_union.args)
      if len(interval_union.boundary)>2:
        merged_tuplist = [tuple(t.boundary) for t in interval_union.args]
      else:
        merged_tuplist = [tuple(interval_union.args)]
      return merged_tuplist
    else:
      return tuplist

def get_blurbs(npset,max_distance=3,known_concepts = [],max_idf = 15):

    tuplistdfall = pd.DataFrame()
    for doc in Agg_sent_to_clt:
      blurblist = []
      tuplist = calc_clt_blurb_order(get_sentence_indices(npset,doc,max_distance),doc,known_concepts,max_idf)
      sents = Agg_sents[doc]
      tuplistdf = pd.DataFrame()
      tuplistindices = list(tuplist.index)
      for t in tuplistindices:
          blurb = ''
          #print(t)x
          for s in range(t[0],t[1]+1):
            blurb = blurb + sents[s]+' '
          #print(blurb)
          blurblist.append(blurb)

      tuplistdf['tuples'] = tuplistindices
      tuplistdf['clt'] = list(tuplist)
      tuplistdf['blurbs'] = blurblist
      tuplistdf['doc'] = doc
      tuplistdfall = tuplistdfall.append(tuplistdf,ignore_index=True)

    return tuplistdfall.sort_values(by='clt',ascending=True)

def flatten_list(listoflists):
  flatlist = []
  for l in listoflists:
    flatlist = flatlist+l
  return flatlist

def findall_string(blurb,np):
  blurb = blurb.lower()
  start = 0
  poslist = []
  flag = 0
  while flag == 0:
    try:
      pos = blurb.index(np,start)
      #print(start,'found')
      poslist.append((pos,pos+len(np)))
      start = pos+len(np)
    except:
      flag = 1
  return poslist

def change_color(blurb,npset):
  #print(npset)
  if len(npset)>1:
    tuplist = flatten_list([findall_string(blurb,np) for np in npset])
  else:
    tuplist = findall_string(blurb,list(npset)[0])
  starts = [t[0] for t in tuplist]
  starts.sort()
  stops = [t[1] for t in tuplist] # making assumptions of no overlap here.
  stops.sort()
  newblurb = "\033[1;0m "+blurb[0:starts[0]]
  for i in range(len(starts)-1):
    newblurb = newblurb + "\033[1;32m "+blurb[starts[i]:stops[i]]
    newblurb = newblurb + "\033[1;0m "+blurb[stops[i]:starts[i+1]]
  newblurb = newblurb + "\033[1;32m "+blurb[starts[-1]:stops[-1]]
  newblurb = newblurb + "\033[1;0m "+blurb[stops[-1]:-1]
  return newblurb

```

```
list(subG.nodes)
```

```
import textwrap

npset = set(['envoy listener',
 #'activation state',
 #'connection oriented tcp filter',
 #'tcp connection',
 #'alpn',
 #'combination',
 'active / passive health check',
 'circuit breaker',
 #'connection',
 'balancing',
 #'activation'
 ])
#print(npset)
tuplistdfall =  get_blurbs(npset,4,known_concepts = [],max_idf = 10)
printable = ''
blurb_count = 1
for i,row in tuplistdfall.iterrows():
  print('Blurb: ',blurb_count)
  print('Document: ',row['doc'],' index: ',row['tuples'])
  printable = change_color(row['blurbs'],npset)
  print(textwrap.fill(printable, 120))
  print('\n\n\n')
  blurb_count +=1

# Should show number of results including each term.

```

    [79] [196]
    [196] [48, 97, 199, 201, 203]
    [79] [48, 97, 199, 201, 203]
    [79] [96, 188, 194, 197]
    [96, 188, 194, 197] [48, 97, 199, 201, 203]
    [196] [96, 188, 194, 197]
    [] []
    [] []
    [] []
    [] []
    [] []
    [] []
    Blurb:  1
    Document:  Envoy_loar  index:  (96, 98)
     Cluster specific load  balancing  is performed to find an endpoint. The cluster’s  circuit
    breaker s are checked to determine if a new stream is allowed. A new connection to the endpoint is created if the
    endpoint’s connection pool is empty or lacks capacity.

    Blurb:  2
    Document:  Envoy_loar  index:  (194, 200)
     Envoy supports a variety of load  balancing  algorithms, e.g. weighted round-robin, Maglev, least-
    loaded, random. Load balancers obtain their effective assignments from a combination of static bootstrap configuration,
    DNS, dynamic xDS (the CDS and EDS discovery services) and active/passive health checks. Further details on how load
     balancing  works in Envoy are provided in the load  balancing  documentation. Once an
    endpoint is selected, the connection pool for this endpoint is used to find a connection to forward the request on. If
    no connection to the host exists, or all connections are at their maximum concurrent stream limit, a new connection is
    established and placed in the connection pool, unless the  circuit breaker  for maximum connections for the
    cluster has tripped. If a maximum lifetime stream limit for a connection is configured and reached, a new connection is
    allocated in the pool and the affected HTTP/2 connection is drained.

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:27: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

Need for proper noun entity resolution

```
All_df.loc[All_df['Concept1'].isin(list(Agg_phen_to_sent.keys())) & All_df['Concept2'].isin(list(Agg_phen_to_sent.keys()))].sort_values(by = ['Raw_TFA','Raw_TFB','phen_fracA','phen_fracB'],ascending=[False]*4).head(10)
```

    Warning: Total number of columns (27) exceeds max_columns (20). Falling back to pandas display.

      Concept1
      Concept2
      Bondstrength
      mean_dAB
      mean_dBA
      ExtIDFA
      ExtIDFB
      SdevA
      SdevB
      Raw_TFA
      Ext_TFIDFA
      Raw_TFB
      Ext_TFIDFB
      MCForm1
      MCForm2
      phen_fracA
      phen_fracB
      att_fracA
      att_fracB
      set_fracA
      set_fracB
      ins_fracA
      ins_fracB
      Envoy_loar_IntTFIDFA
      Envoy_loar_IntTFIDFB
      TLSwiki_IntTFIDFA
      TLSwiki_IntTFIDFB

      5698
      connection
      request
      0.200000
      5.551724
      9.687500
      1.184634
      1.177845
      0.211875
      0.339466
      47
      55.677791
      38
      44.758125
      connection
      request
      0.348214
      0.283951
      0.348214
      0.283951
      0.303571
      0.432099
      0.0
      0.0
      -16.963913
      -18.718800
      -10.529325
      -3.509775

      5691
      connection
      handshake
      0.833333
      11.444444
      6.466667
      1.184634
      5.519732
      0.335370
      0.364488
      47
      55.677791
      20
      110.394634
      connection
      handshake
      0.348214
      0.403846
      0.348214
      0.403846
      0.303571
      0.192308
      0.0
      0.0
      -16.963913
      -2.924813
      -10.529325
      -8.774438

      5700
      connection
      response
      0.200000
      6.379310
      7.312500
      1.184634
      0.864783
      0.211875
      0.280271
      47
      55.677791
      16
      13.836533
      connection
      response
      0.348214
      0.372093
      0.348214
      0.372093
      0.303571
      0.255814
      0.0
      0.0
      -16.963913
      0.000000
      -10.529325
      0.000000

      5706
      connection
      session
      0.200000
      29.222222
      14.000000
      1.184634
      1.588606
      0.335370
      0.208000
      47
      55.677791
      13
      20.651877
      connection
      session
      0.348214
      0.371429
      0.348214
      0.371429
      0.303571
      0.257143
      0.0
      0.0
      -16.963913
      0.000000
      -10.529325
      0.000000

      5688
      connection
      extension
      0.250000
      42.827586
      12.500000
      1.184634
      1.597636
      0.211875
      0.324597
      47
      55.677791
      9
      14.378725
      connection
      extension
      0.348214
      0.312500
      0.348214
      0.312500
      0.303571
      0.375000
      0.0
      0.0
      -16.963913
      -1.169925
      -10.529325
      -4.094738

      5715
      connection
      tl session
      0.200000
      48.944444
      13.000000
      1.184634
      5.561526
      0.335370
      0.268572
      47
      55.677791
      6
      33.369155
      connection
      TLS session
      0.348214
      0.111111
      0.348214
      0.111111
      0.303571
      0.777778
      0.0
      0.0
      -16.963913
      0.000000
      -10.529325
      0.000000

      5723
      connection
      upstream connection
      0.500000
      21.517241
      3.333333
      1.184634
      3.410212
      0.211875
      0.249378
      47
      55.677791
      3
      10.230635
      connection
      upstream connection
      0.348214
      0.285714
      0.348214
      0.285714
      0.303571
      0.428571
      0.0
      0.0
      -16.963913
      0.000000
      -10.529325
      0.000000

      5719
      connection
      transport security
      0.333333
      77.000000
      2.000000
      1.184634
      1.183904
      0.211875
      0.000000
      47
      55.677791
      1
      1.183904
      connection
      transport security
      0.348214
      0.500000
      0.348214
      0.500000
      0.303571
      0.000000
      0.0
      0.0
      -16.963913
      0.000000
      -10.529325
      0.000000

      5721
      connection
      transport socket decryption
      0.333333
      77.000000
      2.000000
      1.184634
      7.027736
      0.211875
      0.000000
      47
      55.677791
      1
      7.027736
      connection
      transport socket decryption
      0.348214
      0.500000
      0.348214
      0.500000
      0.303571
      0.000000
      0.0
      0.0
      -16.963913
      0.000000
      -10.529325
      0.000000

      22603
      request
      connection
      0.333333
      4.333333
      6.500000
      1.177845
      1.184634
      0.245548
      0.335370
      38
      44.758125
      47
      55.677791
      request
      connection
      0.283951
      0.348214
      0.283951
      0.348214
      0.432099
      0.303571
      0.0
      0.0
      -18.718800
      -16.963913
      -3.509775
      -10.529325

      <button class="colab-df-convert" onclick="convertToInteractive('df-c5d7f552-947a-4821-9993-fde4bba56078')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

Phenomena graphic organizer