**Todo**
---
> 1. Make all the Conv Layers functions <-- did this
> 2. 2x2 inputs and 4x4 inputs <-- is finished ^
> 3. Adam Optimizer <-- not important right now
> 4. 1.6 in tutorial <-- is finished
> 5. 1.7 in tutorial <-- Saturday

**Installation of Necessary Modules**
1.   Tensorflow==2.4.1
2.   Tensorflow-Quantum
3.   Sympy
4.   Numpy
5.   Google Cirq
6.   Pennylane (Just in Case)

```python
!pip install -q tensorflow==2.4.1 tensorflow-quantum
!pip install cirq sympy numpy pennylane
```

**Importing Modules**

1.   Matplotlib.pyplot as plt
2.   SVGCircuit from Cirq.contrib.svg
3.   Tensorflow as tf and Tensorflow_Quantum as tfq
4.   Google Cirq for writing the circuit
5.   Sympy as sp
6.   Numpy as np from Pennylane
7.   Pennylane as qml
8.   Itertools for efficient iteration
9.   Math for doing math

```python
#%tb #full traceback for errors
%matplotlib inline
import tensorflow as tf
import tensorflow_quantum as tfq
import matplotlib.pyplot as plt
import cirq
from cirq.contrib.svg import SVGCircuit
import sympy as sp
import pennylane
from pennylane import numpy as np
from math import *
from itertools import *
```

**Abhik's Section**

```python
#Can put code here
```

**Function which Creates the Circuit and Qubits**

```python
numOfQubits = 4
```

```python
def createCircuit(num_qubits):
    qubits = cirq.GridQubit.rect(1, num_qubits)
    circuit = cirq.Circuit()
    return qubits, circuit
```

```python
qubits, vqc = createCircuit(numOfQubits)
```

*If you mess up run the "I messed up function"*

```python
#I messed up function
#qubits, vqc = createCircuit(4)
```

**Cluster State PQC**
1. Hadamard all Qubit
2. CNOT each pair of qubits (0,1), (1,2) (2,3), (3,0)

```python
def Cluster_State_PQC(bits):
    circuit = cirq.Circuit()
    circuit.append(cirq.H.on_each(bits))
    for this_bit, next_bit in zip(bits, bits[1:] + [bits[0]]):
        circuit.append(cirq.CZ(this_bit, next_bit))
    return circuit
```

**Rotation Angle Initialization before perfoming Convolution and Pooling and Gate Functions**

```python
def InitRot(numOfAngles=74, angle = np.pi/2):
    return np.array([angle for i in range(numOfAngles)])
```

```python
default = InitRot()
print(default)
```

    [1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633 1.57079633 1.57079633 1.57079633 1.57079633
     1.57079633 1.57079633]

*Rotation angles subject to changes at any time*

```python
def U3_Gate(bit, params):
    return cirq.Circuit(
        cirq.X(bit)**params[0],
        cirq.Y(bit)**params[1],
        cirq.Z(bit)**params[2])
```

```python
def CU_Gate(other_bit, bit, params):
    return cirq.Circuit(
        cirq.X(bit).controlled_by(other_bit)**params[0],
        cirq.Y(bit).controlled_by(other_bit)**params[1],
        cirq.Z(bit).controlled_by(other_bit)**params[2])
```

**Quantum Convolution Layer 1**.
*Consists of RY gates with fixed angle of rotation (non-trainable) and CNOT pattern as in the PQC*

```python
def QConvLayer1(qubits):
    circuit = cirq.Circuit()
    for first in qubits:
        circuit.append(cirq.Ry(rads=np.pi)(first))
    for first, second in zip(qubits[0::1], qubits[1::1] + [qubits[0]]):
        circuit.append(cirq.CNOT(first, second))
    return circuit
```

**Quantum Convolution Layer 2**
*Consists of U3 gates applied to each qubit and CU gates applied in the CNOT pattern*

```python
def QConvLayer2(qubits, params):
    circuit = cirq.Circuit()
    index = 0
    for qubit in qubits:
        circuit += U3_Gate(qubit, params[index : index+3])
        index += 3
    for first, second in zip(qubits[0::1], qubits[1::1] + [qubits[0]]):
        circuit += CU_Gate(first, second, params[index : index+3])
        index += 3
    return circuit
```

**Quantum Convolution Layer 3**

```python
def QConvLayer3(qubits, params):
    circuit = cirq.Circuit()
    index=0
    for qubit in qubits:
        circuit += U3_Gate(qubit, params[index : index+3])
        index += 3
    circuit += cirq.CNOT(control=qubits[0], target=qubits[1])
    circuit += (cirq.CNOT(control=qubits[3], target=qubits[2]))
    circuit += U3_Gate(qubits[0], [np.pi, np.pi, np.pi])
    circuit += U3_Gate(qubits[3], [np.pi/2, np.pi/2, np.pi/2])
    circuit += cirq.CNOT(control=qubits[0], target=qubits[1])
    circuit += cirq.CNOT(control=qubits[3], target=qubits[2])
    return circuit
```

**Quantum Convolution Layer 4**

```python
def QConvLayer4(qubits, params):
    circuit = cirq.Circuit()
    index = 0
    circuit += U3_Gate(qubits[0], params[0:3])
    for first, second in zip(qubits[0::1], qubits[1::1] + [qubits[0]]):
        circuit += cirq.CNOT(control=first, target=second)
        circuit += CU_Gate(first, second, params[index:index+3])
        index += 3
    return circuit
```

**Quantum Convolution Layer 5**

```python
def QConvLayer5(qubits, params):
    circuit = cirq.Circuit()
    circuit += U3_Gate(qubits[0], params[0:3])
    circuit += cirq.CNOT(control=qubits[0], target=qubits[1])
    circuit += cirq.CNOT(control=qubits[2], target=qubits[3])
    circuit += U3_Gate(qubits[2], params[3:6])
    circuit += cirq.CNOT(control = qubits[2], target=qubits[1])
    circuit += U3_Gate(qubits[1], params[6:9])
    return circuit
```

**Quantum Convolution Layer 6**
*Same as Layer 4 but CU is CNOT*

```python
def QConvLayer6(qubits):
    circuit = cirq.Circuit()
    for i in range(1, len(qubits)):
        circuit += cirq.CNOT(qubits[0], qubits[i])
    circuit += U3_Gate(qubits[3], [np.pi, np.pi/2, np.pi/4])
    return circuit
```

**Quantum Pooling Layer**

```python
def QPoolLayer(source, sink, params):
    poolCircuit = cirq.Circuit()
    sinkBasisSelector = U3_Gate(sink, params[0:3])
    sourceBasisSelector = U3_Gate(source, params[3:6])
    poolCircuit.append(sinkBasisSelector)
    poolCircuit.append(sourceBasisSelector)
    poolCircuit.append(cirq.CNOT(control=source, target=sink))
    poolCircuit.append(sinkBasisSelector**-1)
    return poolCircuit
```

**1.6 Create Model**

```python
def createModelCircuit(qubits, symbols=sp.symbols('param0:74')):
    modelCircuit = cirq.Circuit()
    modelCircuit += QConvLayer1(qubits)
    modelCircuit += QConvLayer2(qubits, symbols[0:24])
    modelCircuit += QConvLayer3(qubits, symbols[24:36])
    modelCircuit += QPoolLayer(qubits[0], qubits[2], symbols[36:42])
    modelCircuit += QConvLayer4(qubits, symbols[42:57])
    modelCircuit += QConvLayer5(qubits, symbols[57:67])
    modelCircuit += QPoolLayer(qubits[1], qubits[3], symbols[67:74])
    modelCircuit += QConvLayer6(qubits)
    return modelCircuit
```

```python
clusterStateBits = cirq.GridQubit.rect(1, 4)
readoutOps = cirq.Z(clusterStateBits[-1])
excitationInput = tf.keras.Input(shape=(), dtype=tf.dtypes.string)
clusterState = tfq.layers.AddCircuit()(excitationInput, prepend=Cluster_State_PQC(clusterStateBits))
QModel = tfq.layers.PQC(createModelCircuit(clusterStateBits), readoutOps)(clusterState)
QCNNModel = tf.keras.Model(inputs=[excitationInput], outputs=[QModel])
tf.keras.utils.plot_model(QCNNModel, show_shapes=True, show_layer_names=True, dpi=70)
```


```javascript
%%javascript
$('#appmode-leave').hide();
$('#copy-binder-link').hide();
$('#visit-repo-link').hide();
```

    <IPython.core.display.Javascript object>

# Bomb Calorimetry

### Video credit: Chris Ambidge - University of Toronto.

```python
# from IPython.display import YouTubeVideo
# # Video credit: Chris Ambidge - University of Toronto.
# YouTubeVideo('bm3Tn6DXJrI')
```

## Introduction to the experiment

**Calorimetry** derives from the Latin word _calor_ meaning heat, and the Greek _metry_ meaning to measure. Thus, it is the science of measuring amounts of heat.
All calorimetric techniques are based on the measurement of heat generated or consumed by a sample, depending on whether exothermic or endothermic processes are involved.
A calorimeter can be a simple Styrofoam cup, because it is a container with insulated walls that prevents heat exchange with the environment, or more a complicated device such as a bomb calorimeter

 Figure 1: Schematic representation of a bomb calorimeter (Image taken taken from chem.libretexts.org)

In the simplest version of this experiment, in order to measure heats of reactions, we enclose the reactants in a calorimeter, allow the reaction to proceed, and measure the temperature change ($\Delta$T). Then the heat released or adsorbed in the reaction can be calculated.
A calorimeter may be used under constant pressure (atmospheric), or constant volume.
Regardless of the setup, the calorimeter adsorbs some of the released heat and its heat capacity must be measured to calibrate the instrument, which is usually done just before or after the actual measurement of the heats of reaction.
This is done by transferring a known amount of heat into the calorimeter and measuring its temperature increase.
Because the temperature differences are often very small, sensitive thermometers are preferred for these measurements.

## Theory
Conventionally, in thermodynamics we use two fundamental definitions: the system and the surroundings.
The system is what is being measured, observed and investigated while the surroundings are everything else.
An isolated system is referred to as adiabatic, which means that there is no exchange of matter or energy between the system and its surroundings.
The energy considerations of a reaction include heat (denoted $q$), which is the energy absorbed by the system, and the work done on the system (denoted $w$).
As heat is transferred into or out of a system this leads to a change in temperature, and the quantity of energy that causes a temperature change is determined with this equation

\begin{equation}
q = \sum c(T_f-T_i) \tag{1}
\end{equation}

Where $q$ is the heat of the process, $\sum c$ is sum of the heat capacities of all the various components of the system, $T_f$ is the final temperature and $T_i$ is the initial temperature. Because it is difficult, if not impossible to separate the different contributions, in the following we will just use an _effective_ heat capacity of the calorimeter, $C_V = \sum c$.
Recalling the physical chemistry lectures, the heat capacity is the amount of heat required to raise the temperature of a substance (or of the entire calorimeter) by 1 K.

### Obtaining an Isothermal Enthalpy
The enthalpy of any process is the heat absorbed by the system at constant pressure.
The enthalpy of a reaction under isothermal conditions can be determined by conducting the experiment adiabatically, and then measuring the temperature change that occurs in the reaction vessel.
The isothermal reaction at temperature $T_i$ can be expressed as follows:

\begin{equation}
\mathrm{A(T_i)} + \mathrm{B(T_i)} \xrightarrow{\Delta H_c} \mathrm{C(T_i)} + \mathrm{D(T_i)} \tag{2}
\end{equation}

Where $\Delta H$ is the enthalpy of the isothermal reaction for each mole of reaction as written.
In a calorimeter the system is insulated, and therefore no energy is lost to the surroundings.
This is known as an adiabatic system and therefore in the calorimeter the energy of the
reaction is used to heat the system with no loss.
The adiabatic reaction can be expressed as follows:

\begin{equation}
\mathrm{A(T_i)} + \mathrm{B(T_i)} + \mathrm{S(T_i)}\xrightarrow{\Delta H_I} \mathrm{C(T_f)} + \mathrm{D(T_f)} + \mathrm{S(T_f)} \tag{3}
\end{equation}

Where $\Delta H_I$ is the enthalpy of the adiabatic reaction.
A, B, C and D continue to be the reactants and products respectively and S is the parts of the system (thermometer, solvent, reaction vessel walls etc.) that maintain the same temperature as the reactants and products because of the insulated system.
Enthalpy however is a state function, and therefore independent of the steps taken to get to the final stage. Thus, the enthalpy of the isothermal reaction will be the sum of the enthalpy of any steps that start and finish in the same state.
Therefore in order to finish the adiabatic reaction under the same conditions as the isothermal reaction, the system merely needs to be cooled to the initial temperature.

\begin{equation}
\mathrm{A(T_f)} + \mathrm{B(T_f)} + \mathrm{S(T_f)}\xrightarrow{\Delta H_{II}} \mathrm{C(T_i)} + \mathrm{D(T_i)} + \mathrm{S(T_i)} \tag{4}
\end{equation}

$\Delta H_{I}$ and $\Delta H_{II}$ are the enthalpies of the alternate steps taken to go from the initial stage to the final stage of the isothermal reaction, and therefore, as shown in the figure below, the enthalpy of the isothermal reaction is the sum of these two values.

\begin{equation}
\Delta H_c = \Delta H_{I} + \Delta H_{II} \tag{5}
\end{equation}

As stated previously an adiabatic reaction is a reaction contained in an isolated system, i.e. no energy (heat) is transferred out of the system.
This means that for the first step in Figure 2, $\mathrm{d}q$=0 and $\Delta H=0$ adiabatic step is 0, and therefore $\Delta H_{I}$ is also 0.
The cooling step indicated in Figure 1 need not be performed, but rather the heat of Step II can be calculated as

\begin{equation}
\Delta H_{II} = C_V (T_f - T_i ) \tag{6}
\end{equation}

 Figure 2. Schematic representation of the working principles of the calorimeter and key equations.

### Reaction Enthalpy Under Standard Conditions
Just as the isothermal enthalpy can be calculated from results that take a different series of steps, the enthalpy of a reaction under standard conditions can be determined from any isothermal energy of that reaction.
An alternative series of steps for the standard reaction to occur is: i) changing the system temperature from standard conditions to the experimental temperature, ii) proceeding with the reaction, iii) changing the temperature back to standard temperature.
As these three steps start with the reactants under standard conditions and finish with the products under standard conditions the sum of the enthalpies of these steps is equivalent to the enthalpy of the reaction proceeding under standard conditions. This is presented as

 Figure 3

## Bomb calorimetry, the experiment
In this *virtual* experiment you will calibrate the calorimeter using benzoic acid and determine the enthalpy of combustion of sucrose and napthalene.

An adiabatic bomb calorimeter provides a constant volume environment for the measurement of the heat of combustion of compounds.
This corresponds to the first step in Figure 2.

As mentioned before, and important step in any calorimetry experiment is to calibrate the instrument, in order to achieve that you will perform a first experiment with benzoic acid, for which you are given its enthalpy of combustion ($\Delta H_c=-3228\ \mathrm{kJ/mol}$).

This reaction proceeds as follows within the calorimeter:

\begin{equation}
\mathrm{C_6H_5COOH} + \frac{15}{2}\mathrm{O_2(excess)} \xrightarrow{20\ atm} \mathrm{7CO_2(g)} + \mathrm{3H_2O(l)} \tag{8}
\end{equation}

The process is carried out under 20 atm of excess oxygen in order to ensure complete and rapid combustion.

These conditions are not exactly standard temperature nor pressure, and therefore the energy change measured is not the standard heat of reaction.
Corrections would need to be made to convert to standard conditions, however the experimental uncertainty in this lab accounts for the errors introduced by the non-standard conditions.
The change in internal energy of the system is defined in the following fashion:

\begin{equation}
\Delta U = dq - P\Delta V \tag{9}
\end{equation}

However, because the experiment is performed a constant volume this becomes

\begin{equation}
\Delta U = dq_V \tag{10}
\end{equation}

Therefore, by using the calorimeter to measure the change in temperature you can measure the change in internal energy of the system

\begin{equation}
\Delta U = dq_V = C_V(T_f-T_i) \tag{11}
\end{equation}

Where $C_V$ is the heat capacity of the entire system at constant volume.
Using the definition of enthalpy

\begin{equation}
H = U + PV \tag{12}
\end{equation}

we can then write the combustion enthalpy change for a reaction as

\begin{equation}
\Delta H_c = \Delta U + \Delta (PV) = \Delta U + V\Delta P + P\Delta V \tag{13}
\end{equation}

At constant volume, because $P\Delta V=0$, this becomes

\begin{equation}
\Delta H_c = \Delta U + \Delta (nRT) \tag{14}
\end{equation}

where we have assumed that the gas is ideal and $V\Delta P=\Delta (nRT)$.
Where $P$ is the pressure in kPa, $V$ is the volume in litres, $\Delta n$ is the change in the number of moles of gas particles during the reaction, $R$ is the universal gas constant and $T$ is the isothermal temperature.

The enthalpy of the combustion reaction in the bomb calorimeter can then be calculated using change of internal energy ($\Delta U=C_v\Delta T$) and from the gas stoichiometry of the equation.

The enthalpy of combustion can also be determined from the formation enthapies of reactants and products

\begin{equation}
\Delta H_c = \sum \Delta H_f^{products} - \sum \Delta H_f^{reactants} \tag{15}
\end{equation}

### Experimental Factors
No experimental configuration is truly adiabatic, and in this case the stirrer adds energy to the system while the thermometer and non-ideal insulation allows heat to escape from the system. These factors produce fairly linear changes in temperature before and after the reaction occurs. In order to accurately account for these effects, the temperature must be measured for a period of time before the reactants are combined and for a period of time after the reaction has proceeded. Plotting this data and recording the relative time the reactants were combined allows the change in temperature to be more accurately determined as shown in the figure below.

 Figure 4

### Virtual experiment procedure
In this virtual lab instead you will have to
1. Choose your sample (Benzoic acid)
2. Create a tablet for you sample
3. "Perform" the experiment (obtain temperature inside the bomb calorimeter vs time)
4. Fit the temperature profile before and after the experiment to obtain the temperature change
5. Determine the calorimeter heat capacity of the bomb calorimeter to calibrate the instrument
6. Repeat 1 to 5 to obtain an average value for the heat capacity of the bomb calorimeter
7. Repeat 1 to 4 for sucrose and naphtalene
8. Determine the combustion enthalpy and formation enthalpy of Sucrose and Napthalene

#### Assume the wire described in the video releases a negligible amount of heat

## Numerical skills

1. fitting of portions of data
2. error analysis

## Pre-lab

1. Obtain from literature the standard enthalpies of formation of $O_2$, $H_2O$, $CO_2$, sucrose and naphtalene.
2. Write the balances combustion reactions for benzoic acid, sucrose and naphtalene
3. Calculate the expected combustion enthalpy of sucrose and naphtalene

## Post-lab
3. Use the given combustion enthalpy of benzoic acid to "calibrate the instrument"; what is the heat capacity of the bomb calorimeter?

4. Calculate the combustion and formation enthalpies ($\Delta H_c$ \& $\Delta H_f$) of sucrose and naphtalene. Compare with the literature values obtained in the pre lab.

## Launch virtual experiment
- [Bomb Calorimetry virtual lab](virtualExperiment.ipynb)


# N-Gram Language Models

Uncomment the code below if you're on Colab

## run code in this section if you're on colab

```python
# !git clone https://github.com/Kabongosalomon/AMMI-NLP.git

# %cd AMMI-NLP/Part\ 02/001_LM/ngram_lm_lab

# !sudo apt-get install build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev

# !wget -O - https://kheafield.com/code/kenlm.tar.gz |tar xz
# !mkdir kenlm/build
# %cd kenlm/build
# !cmake ..
# !make -j 4

# %cd ..

# ! python setup.py install

# %cd ..
```

    /content/AMMI-NLP/Part 02/001_LM/ngram_lm_lab
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    build-essential is already the newest version (12.4ubuntu1).
    liblzma-dev is already the newest version (5.2.2-1.3).
    liblzma-dev set to manually installed.
    zlib1g-dev is already the newest version (1:1.2.11.dfsg-0ubuntu2).
    zlib1g-dev set to manually installed.
    libboost-all-dev is already the newest version (1.65.1.0ubuntu1).
    cmake is already the newest version (3.10.2-1ubuntu2.18.04.1).
    libbz2-dev is already the newest version (1.0.6-8.1ubuntu0.2).
    libbz2-dev set to manually installed.
    0 upgraded, 0 newly installed, 0 to remove and 25 not upgraded.
    --2020-03-21 00:54:59--  https://kheafield.com/code/kenlm.tar.gz
    Resolving kheafield.com (kheafield.com)... 35.196.63.85
    Connecting to kheafield.com (kheafield.com)|35.196.63.85|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 490005 (479K) [application/x-gzip]
    Saving to: ‘STDOUT’

    -                   100%[===================>] 478.52K  1007KB/s    in 0.5s

    2020-03-21 00:54:59 (1007 KB/s) - written to stdout [490005/490005]

    /content/AMMI-NLP/Part 02/001_LM/ngram_lm_lab/kenlm/build
    -- The C compiler identification is GNU 7.5.0
    -- The CXX compiler identification is GNU 7.5.0
    -- Check for working C compiler: /usr/bin/cc
    -- Check for working C compiler: /usr/bin/cc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Looking for pthread.h
    -- Looking for pthread.h - found
    -- Looking for pthread_create
    -- Looking for pthread_create - not found
    -- Looking for pthread_create in pthreads
    -- Looking for pthread_create in pthreads - not found
    -- Looking for pthread_create in pthread
    -- Looking for pthread_create in pthread - found
    -- Found Threads: TRUE
    -- Boost version: 1.65.1
    -- Found the following Boost libraries:
    --   program_options
    --   system
    --   thread
    --   unit_test_framework
    --   chrono
    --   date_time
    --   atomic
    -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
    -- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version "1.0.6")
    -- Looking for BZ2_bzCompressInit
    -- Looking for BZ2_bzCompressInit - found
    -- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so
    -- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found
    -- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so
    -- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found
    -- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so
    -- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so - found
    -- Found LibLZMA: /usr/include (found version "5.2.2")
    -- Could NOT find Eigen3 (missing: EIGEN3_INCLUDE_DIR EIGEN3_VERSION_OK) (Required is at least version "2.91.0")
    CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
      Not building interpolation.  Eigen3 was not found.

    -- To install Eigen3 in your home directory, copy paste this:
    export EIGEN3_ROOT=$HOME/eigen-eigen-07105f7124f9
    (cd $HOME; wget -O - https://bitbucket.org/eigen/eigen/get/3.2.8.tar.bz2 |tar xj)
    rm CMakeCache.txt

    -- Configuring done
    -- Generating done
    -- Build files have been written to: /content/AMMI-NLP/Part 02/001_LM/ngram_lm_lab/kenlm/build
    Scanning dependencies of target kenlm_filter
    Scanning dependencies of target kenlm_util
    [  1%] Building CXX object lm/filter/CMakeFiles/kenlm_filter.dir/arpa_io.cc.o
    [  2%] Building CXX object lm/filter/CMakeFiles/kenlm_filter.dir/phrase.cc.o
    [  3%] Building CXX object lm/filter/CMakeFiles/kenlm_filter.dir/vocab.cc.o
    [  5%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum-dtoa.cc.o
    [  6%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum.cc.o
    [  7%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/cached-powers.cc.o
    [  8%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/diy-fp.cc.o
    [ 10%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/double-conversion.cc.o
    [ 11%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fast-dtoa.cc.o
    [ 12%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fixed-dtoa.cc.o
    [ 13%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/strtod.cc.o
    [ 15%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/chain.cc.o
    [ 16%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/count_records.cc.o
    [ 17%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/io.cc.o
    [ 18%] Linking CXX static library ../../lib/libkenlm_filter.a
    [ 18%] Built target kenlm_filter
    [ 20%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/line_input.cc.o
    [ 21%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/multi_progress.cc.o
    [ 22%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/rewindable_stream.cc.o
    [ 23%] Building CXX object util/CMakeFiles/kenlm_util.dir/bit_packing.cc.o
    [ 25%] Building CXX object util/CMakeFiles/kenlm_util.dir/ersatz_progress.cc.o
    [ 26%] Building CXX object util/CMakeFiles/kenlm_util.dir/exception.cc.o
    [ 27%] Building CXX object util/CMakeFiles/kenlm_util.dir/file.cc.o
    [ 28%] Building CXX object util/CMakeFiles/kenlm_util.dir/file_piece.cc.o
    [ 30%] Building CXX object util/CMakeFiles/kenlm_util.dir/float_to_string.cc.o
    [ 31%] Building CXX object util/CMakeFiles/kenlm_util.dir/integer_to_string.cc.o
    [ 32%] Building CXX object util/CMakeFiles/kenlm_util.dir/mmap.cc.o
    [ 33%] Building CXX object util/CMakeFiles/kenlm_util.dir/murmur_hash.cc.o
    [ 35%] Building CXX object util/CMakeFiles/kenlm_util.dir/parallel_read.cc.o
    [ 36%] Building CXX object util/CMakeFiles/kenlm_util.dir/pool.cc.o
    [ 37%] Building CXX object util/CMakeFiles/kenlm_util.dir/read_compressed.cc.o
    [ 38%] Building CXX object util/CMakeFiles/kenlm_util.dir/scoped.cc.o
    [ 40%] Building CXX object util/CMakeFiles/kenlm_util.dir/spaces.cc.o
    [ 41%] Building CXX object util/CMakeFiles/kenlm_util.dir/string_piece.cc.o
    [ 42%] Building CXX object util/CMakeFiles/kenlm_util.dir/usage.cc.o
    [ 43%] Linking CXX static library ../lib/libkenlm_util.a
    [ 43%] Built target kenlm_util
    Scanning dependencies of target probing_hash_table_benchmark
    Scanning dependencies of target kenlm
    [ 45%] Building CXX object util/CMakeFiles/probing_hash_table_benchmark.dir/probing_hash_table_benchmark_main.cc.o
    [ 46%] Building CXX object lm/CMakeFiles/kenlm.dir/binary_format.cc.o
    [ 47%] Building CXX object lm/CMakeFiles/kenlm.dir/config.cc.o
    [ 48%] Building CXX object lm/CMakeFiles/kenlm.dir/bhiksha.cc.o
    [ 50%] Building CXX object lm/CMakeFiles/kenlm.dir/lm_exception.cc.o
    [ 51%] Building CXX object lm/CMakeFiles/kenlm.dir/model.cc.o
    [ 52%] Building CXX object lm/CMakeFiles/kenlm.dir/quantize.cc.o
    [ 53%] Building CXX object lm/CMakeFiles/kenlm.dir/read_arpa.cc.o
    [ 55%] Building CXX object lm/CMakeFiles/kenlm.dir/search_hashed.cc.o
    [ 56%] Building CXX object lm/CMakeFiles/kenlm.dir/search_trie.cc.o
    [ 57%] Building CXX object lm/CMakeFiles/kenlm.dir/sizes.cc.o
    [ 58%] Building CXX object lm/CMakeFiles/kenlm.dir/trie.cc.o
    [ 60%] Building CXX object lm/CMakeFiles/kenlm.dir/trie_sort.cc.o
    [ 61%] Building CXX object lm/CMakeFiles/kenlm.dir/value_build.cc.o
    [ 62%] Building CXX object lm/CMakeFiles/kenlm.dir/virtual_interface.cc.o
    [ 63%] Building CXX object lm/CMakeFiles/kenlm.dir/vocab.cc.o
    [ 65%] Linking CXX executable ../bin/probing_hash_table_benchmark
    [ 65%] Built target probing_hash_table_benchmark
    [ 66%] Building CXX object lm/CMakeFiles/kenlm.dir/common/print.cc.o
    [ 67%] Building CXX object lm/CMakeFiles/kenlm.dir/common/model_buffer.cc.o
    [ 68%] Building CXX object lm/CMakeFiles/kenlm.dir/common/renumber.cc.o
    [ 70%] Building CXX object lm/CMakeFiles/kenlm.dir/common/size_option.cc.o
    [ 71%] Linking CXX static library ../lib/libkenlm.a
    [ 71%] Built target kenlm
    Scanning dependencies of target kenlm_benchmark
    Scanning dependencies of target fragment
    Scanning dependencies of target build_binary
    Scanning dependencies of target query
    [ 72%] Building CXX object lm/CMakeFiles/fragment.dir/fragment_main.cc.o
    [ 73%] Building CXX object lm/CMakeFiles/build_binary.dir/build_binary_main.cc.o
    [ 75%] Building CXX object lm/CMakeFiles/kenlm_benchmark.dir/kenlm_benchmark_main.cc.o
    [ 76%] Building CXX object lm/CMakeFiles/query.dir/query_main.cc.o
    [ 77%] Linking CXX executable ../bin/fragment
    [ 77%] Built target fragment
    [ 78%] Linking CXX executable ../bin/build_binary
    Scanning dependencies of target kenlm_builder
    [ 80%] Building CXX object lm/builder/CMakeFiles/kenlm_builder.dir/adjust_counts.cc.o
    [ 80%] Built target build_binary
    Scanning dependencies of target phrase_table_vocab
    [ 81%] Building CXX object lm/filter/CMakeFiles/phrase_table_vocab.dir/phrase_table_vocab_main.cc.o
    [ 82%] Linking CXX executable ../bin/query
    [ 82%] Built target query
    Scanning dependencies of target filter
    [ 83%] Building CXX object lm/filter/CMakeFiles/filter.dir/filter_main.cc.o
    [ 85%] Linking CXX executable ../../bin/phrase_table_vocab
    [ 86%] Building CXX object lm/builder/CMakeFiles/kenlm_builder.dir/corpus_count.cc.o
    [ 86%] Built target phrase_table_vocab
    [ 87%] Building CXX object lm/builder/CMakeFiles/kenlm_builder.dir/initial_probabilities.cc.o
    [ 88%] Building CXX object lm/builder/CMakeFiles/kenlm_builder.dir/interpolate.cc.o
    [ 90%] Building CXX object lm/builder/CMakeFiles/kenlm_builder.dir/output.cc.o
    [ 91%] Building CXX object lm/builder/CMakeFiles/kenlm_builder.dir/pipeline.cc.o
    [ 92%] Linking CXX executable ../bin/kenlm_benchmark
    [ 92%] Built target kenlm_benchmark
    [ 93%] Linking CXX executable ../../bin/filter
    [ 93%] Built target filter
    [ 95%] Linking CXX static library ../../lib/libkenlm_builder.a
    [ 95%] Built target kenlm_builder
    Scanning dependencies of target count_ngrams
    Scanning dependencies of target lmplz
    [ 97%] Building CXX object lm/builder/CMakeFiles/lmplz.dir/lmplz_main.cc.o
    [ 97%] Building CXX object lm/builder/CMakeFiles/count_ngrams.dir/count_ngrams_main.cc.o
    [ 98%] Linking CXX executable ../../bin/lmplz
    [ 98%] Built target lmplz
    [100%] Linking CXX executable ../../bin/count_ngrams
    [100%] Built target count_ngrams
    /content/AMMI-NLP/Part 02/001_LM/ngram_lm_lab/kenlm
    running install
    running bdist_egg
    running egg_info
    creating kenlm.egg-info
    writing kenlm.egg-info/PKG-INFO
    writing dependency_links to kenlm.egg-info/dependency_links.txt
    writing top-level names to kenlm.egg-info/top_level.txt
    writing manifest file 'kenlm.egg-info/SOURCES.txt'
    reading manifest file 'kenlm.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'kenlm.egg-info/SOURCES.txt'
    installing library code to build/bdist.linux-x86_64/egg
    running install_lib
    running build_ext
    building 'kenlm' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/util
    creating build/temp.linux-x86_64-3.6/lm
    creating build/temp.linux-x86_64-3.6/util/double-conversion
    creating build/temp.linux-x86_64-3.6/python
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/integer_to_string.cc -o build/temp.linux-x86_64-3.6/util/integer_to_string.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/parallel_read.cc -o build/temp.linux-x86_64-3.6/util/parallel_read.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/string_piece.cc -o build/temp.linux-x86_64-3.6/util/string_piece.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/float_to_string.cc -o build/temp.linux-x86_64-3.6/util/float_to_string.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/bit_packing.cc -o build/temp.linux-x86_64-3.6/util/bit_packing.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/read_compressed.cc -o build/temp.linux-x86_64-3.6/util/read_compressed.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/spaces.cc -o build/temp.linux-x86_64-3.6/util/spaces.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/murmur_hash.cc -o build/temp.linux-x86_64-3.6/util/murmur_hash.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/file_piece.cc -o build/temp.linux-x86_64-3.6/util/file_piece.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/pool.cc -o build/temp.linux-x86_64-3.6/util/pool.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/scoped.cc -o build/temp.linux-x86_64-3.6/util/scoped.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/exception.cc -o build/temp.linux-x86_64-3.6/util/exception.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/mmap.cc -o build/temp.linux-x86_64-3.6/util/mmap.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/file.cc -o build/temp.linux-x86_64-3.6/util/file.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/usage.cc -o build/temp.linux-x86_64-3.6/util/usage.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/ersatz_progress.cc -o build/temp.linux-x86_64-3.6/util/ersatz_progress.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/vocab.cc -o build/temp.linux-x86_64-3.6/lm/vocab.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/value_build.cc -o build/temp.linux-x86_64-3.6/lm/value_build.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/lm_exception.cc -o build/temp.linux-x86_64-3.6/lm/lm_exception.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/config.cc -o build/temp.linux-x86_64-3.6/lm/config.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/trie.cc -o build/temp.linux-x86_64-3.6/lm/trie.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/binary_format.cc -o build/temp.linux-x86_64-3.6/lm/binary_format.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/model.cc -o build/temp.linux-x86_64-3.6/lm/model.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/sizes.cc -o build/temp.linux-x86_64-3.6/lm/sizes.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/bhiksha.cc -o build/temp.linux-x86_64-3.6/lm/bhiksha.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/read_arpa.cc -o build/temp.linux-x86_64-3.6/lm/read_arpa.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/search_hashed.cc -o build/temp.linux-x86_64-3.6/lm/search_hashed.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/virtual_interface.cc -o build/temp.linux-x86_64-3.6/lm/virtual_interface.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/trie_sort.cc -o build/temp.linux-x86_64-3.6/lm/trie_sort.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/quantize.cc -o build/temp.linux-x86_64-3.6/lm/quantize.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c lm/search_trie.cc -o build/temp.linux-x86_64-3.6/lm/search_trie.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/double-conversion/fixed-dtoa.cc -o build/temp.linux-x86_64-3.6/util/double-conversion/fixed-dtoa.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/double-conversion/strtod.cc -o build/temp.linux-x86_64-3.6/util/double-conversion/strtod.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/double-conversion/bignum-dtoa.cc -o build/temp.linux-x86_64-3.6/util/double-conversion/bignum-dtoa.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/double-conversion/double-conversion.cc -o build/temp.linux-x86_64-3.6/util/double-conversion/double-conversion.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/double-conversion/bignum.cc -o build/temp.linux-x86_64-3.6/util/double-conversion/bignum.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/double-conversion/diy-fp.cc -o build/temp.linux-x86_64-3.6/util/double-conversion/diy-fp.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/double-conversion/cached-powers.cc -o build/temp.linux-x86_64-3.6/util/double-conversion/cached-powers.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c util/double-conversion/fast-dtoa.cc -o build/temp.linux-x86_64-3.6/util/double-conversion/fast-dtoa.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c python/score_sentence.cc -o build/temp.linux-x86_64-3.6/python/score_sentence.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c python/kenlm.cpp -o build/temp.linux-x86_64-3.6/python/kenlm.o -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -std=c++11 -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB
    creating build/lib.linux-x86_64-3.6
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/util/integer_to_string.o build/temp.linux-x86_64-3.6/util/parallel_read.o build/temp.linux-x86_64-3.6/util/string_piece.o build/temp.linux-x86_64-3.6/util/float_to_string.o build/temp.linux-x86_64-3.6/util/bit_packing.o build/temp.linux-x86_64-3.6/util/read_compressed.o build/temp.linux-x86_64-3.6/util/spaces.o build/temp.linux-x86_64-3.6/util/murmur_hash.o build/temp.linux-x86_64-3.6/util/file_piece.o build/temp.linux-x86_64-3.6/util/pool.o build/temp.linux-x86_64-3.6/util/scoped.o build/temp.linux-x86_64-3.6/util/exception.o build/temp.linux-x86_64-3.6/util/mmap.o build/temp.linux-x86_64-3.6/util/file.o build/temp.linux-x86_64-3.6/util/usage.o build/temp.linux-x86_64-3.6/util/ersatz_progress.o build/temp.linux-x86_64-3.6/lm/vocab.o build/temp.linux-x86_64-3.6/lm/value_build.o build/temp.linux-x86_64-3.6/lm/lm_exception.o build/temp.linux-x86_64-3.6/lm/config.o build/temp.linux-x86_64-3.6/lm/trie.o build/temp.linux-x86_64-3.6/lm/binary_format.o build/temp.linux-x86_64-3.6/lm/model.o build/temp.linux-x86_64-3.6/lm/sizes.o build/temp.linux-x86_64-3.6/lm/bhiksha.o build/temp.linux-x86_64-3.6/lm/read_arpa.o build/temp.linux-x86_64-3.6/lm/search_hashed.o build/temp.linux-x86_64-3.6/lm/virtual_interface.o build/temp.linux-x86_64-3.6/lm/trie_sort.o build/temp.linux-x86_64-3.6/lm/quantize.o build/temp.linux-x86_64-3.6/lm/search_trie.o build/temp.linux-x86_64-3.6/util/double-conversion/fixed-dtoa.o build/temp.linux-x86_64-3.6/util/double-conversion/strtod.o build/temp.linux-x86_64-3.6/util/double-conversion/bignum-dtoa.o build/temp.linux-x86_64-3.6/util/double-conversion/double-conversion.o build/temp.linux-x86_64-3.6/util/double-conversion/bignum.o build/temp.linux-x86_64-3.6/util/double-conversion/diy-fp.o build/temp.linux-x86_64-3.6/util/double-conversion/cached-powers.o build/temp.linux-x86_64-3.6/util/double-conversion/fast-dtoa.o build/temp.linux-x86_64-3.6/python/score_sentence.o build/temp.linux-x86_64-3.6/python/kenlm.o -lstdc++ -lrt -lz -lbz2 -llzma -o build/lib.linux-x86_64-3.6/kenlm.cpython-36m-x86_64-linux-gnu.so
    creating build/bdist.linux-x86_64
    creating build/bdist.linux-x86_64/egg
    copying build/lib.linux-x86_64-3.6/kenlm.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
    creating stub loader for kenlm.cpython-36m-x86_64-linux-gnu.so
    byte-compiling build/bdist.linux-x86_64/egg/kenlm.py to kenlm.cpython-36.pyc
    creating build/bdist.linux-x86_64/egg/EGG-INFO
    copying kenlm.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
    copying kenlm.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
    copying kenlm.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
    copying kenlm.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
    writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
    zip_safe flag not set; analyzing archive contents...
    __pycache__.kenlm.cpython-36: module references __file__
    creating dist
    creating 'dist/kenlm-0.0.0-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
    removing 'build/bdist.linux-x86_64/egg' (and everything under it)
    Processing kenlm-0.0.0-py3.6-linux-x86_64.egg
    creating /usr/local/lib/python3.6/dist-packages/kenlm-0.0.0-py3.6-linux-x86_64.egg
    Extracting kenlm-0.0.0-py3.6-linux-x86_64.egg to /usr/local/lib/python3.6/dist-packages
    Adding kenlm 0.0.0 to easy-install.pth file

    Installed /usr/local/lib/python3.6/dist-packages/kenlm-0.0.0-py3.6-linux-x86_64.egg
    Processing dependencies for kenlm==0.0.0
    Finished processing dependencies for kenlm==0.0.0
    /content/AMMI-NLP/Part 02/001_LM/ngram_lm_lab

```python
!ls
```

    data  img  kenlm  models  ngram_lms.ipynb

Please, restard runtime before running the code bellow

#### Preliminaries: Install KenLM

[KenLM](https://kheafield.com/code/kenlm/) is used in the latter part of this lab.

To install it, run the following commands **in the jupyter terminal** (see screenshot):

    sudo apt-get install build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev

    cd /home/jupyter
    wget -O - https://kheafield.com/code/kenlm.tar.gz |tar xz
    mkdir kenlm/build
    cd kenlm/build
    cmake ..
    make -j 4

    cd /home/jupyter/kenlm
    python setup.py install

```python
# %cd AMMI-NLP/Part\ 02/001_LM/ngram_lm_lab
!pip install jsonlines
```

    Requirement already satisfied: jsonlines in /home/aims/anaconda3/envs/aims/lib/python3.7/site-packages (1.2.0)
    Requirement already satisfied: six in /home/aims/anaconda3/envs/aims/lib/python3.7/site-packages (from jsonlines) (1.12.0)

```python
%pylab inline
import os, sys, glob, json, math
import numpy as np
import pandas as pd
from tqdm import tqdm
from pprint import pprint
from collections import defaultdict
pd.set_option('display.max_colwidth', -1)
```

    Populating the interactive namespace from numpy and matplotlib

    /home/aims/anaconda3/envs/aims/lib/python3.7/site-packages/IPython/core/magics/pylab.py:160: UserWarning: pylab import has clobbered these variables: ['f']
    `%matplotlib` prevents importing * from pylab and numpy
      "\n`%matplotlib` prevents importing * from pylab and numpy"

### N-gram Language Modeling

In **language modeling** we want to model the probability of variable length sequences, $$\large p(w_1,\ldots,w_T)=\prod_{t=1}^T p(w_t|w_{<t}).$$

An **n-gram language model** assumes that each word $w_t$ only depends on the preceding $n-1$ words, $$\large p(w_1,\ldots,w_T)=\prod_{t=1}^T p(w_t|w_{t-n+1},\ldots,w_{t-1}).$$

#### Example
For instance, when modeling the sentence $$\texttt{the cat sat on the mat .}$$ a 3-gram language model assumes that $$p(\texttt{mat}|\texttt{the cat sat on the}) \approx p(\texttt{mat}|\texttt{on the}).$$

The sub-sequence $(\texttt{on the mat})$ is a *3-gram* or *trigram*.

### Count-based Estimation

Given some dataset $D$ of sequences, we can estimate an n-gram model through counting, derived as follows:

\begin{align}
p(w_t|w_{t-n+1},\ldots,w_{t-1}) &= \frac{p(w_{t-n+1},\ldots,w_t)}{p(w_{t-n+1},\ldots,w_{t-1})} & \text{definition of conditional probability}\\
                       &= \frac{p(w_{t-n+1},\ldots,w_t)}{\sum_{w_{t'}}p(w_{t-n+1},\ldots,w_{t-1},w_{t'})}\\
                       &\approx \frac{\frac{1}{N}\text{count}(w_{t-n+1},\ldots,w_t)}{\frac{1}{N}\sum_{w_{t'}}\text{count}(w_{t-n+1},\ldots,w_{t-1},w_{t'})}\\
                       &= \frac{\text{count}(w_{t-n+1},\ldots,w_t)}{\sum_{w_{t'}}\text{count}(w_{t-n+1},\ldots,w_{t-1},w_{t'})}\\
                       &= \frac{\text{count}(w_{t-n+1},\ldots,w_t)}{\text{count}(w_{t-n+1},\ldots,w_{t-1})},
\end{align}

where $N$ is the number of $n$-grams in the dataset.

In Python, we can collect these counts into a dictionary mapping a prefix to a dictionary of counts:

        count[(w_n+1,...,w_t-1)] = {wt1: count of (w_n+1,...,w_t1),
                                    wt2: count of (w_n+1,...,w_t2),
                                    ...
                                   }

and for the denominator, maintain a dictionary of totals:

        total[(w_n+1,...,w_t-1)] = count of w_n+1,...,w_t-1

### Simplified Language

To get intuition, lets start by modeling a simple language called `ABC`. A word in this language is one of three tokens, $$w\in \{\texttt{A, B, C}\},$$
and we'll denote a sentence as $\textbf{w}=(w_1,\ldots,w_{|\textbf{w}|})$.

Suppose we are given the following dataset, and want to estimate a **bigram model**: $$p(\textbf{w})\approx\prod_{t=1}^{|\textbf{w}|}p(w_t|w_{t-1})\quad\quad(*)$$

```python
data_raw = [['A', 'A', 'B', 'B'],
            ['A', 'A', 'B'],
            ['A', 'A', 'B', 'C'],
            ['A', 'A', 'A'],
            ['A', 'A', 'A', 'A']]
```

Since our model is a probability distribution, the total probability of all possible strings in the language must sum to 1, i.e.: $$\sum_{\textbf{w}}p(\textbf{w})=1.$$

In order to satisfy this criterion it turns out that we need an additional **beginning token**, `<bos>`, and **end token**, `<eos>`:

```python
data = [['<bos>'] + d + ['<eos>'] for d in data_raw]
data
```

    [['<bos>', 'A', 'A', 'B', 'B', '<eos>'],
     ['<bos>', 'A', 'A', 'B', '<eos>'],
     ['<bos>', 'A', 'A', 'B', 'C', '<eos>'],
     ['<bos>', 'A', 'A', 'A', '<eos>'],
     ['<bos>', 'A', 'A', 'A', 'A', '<eos>']]

Now let's estimate a bigram model:

\begin{align}
p(w_t|w_{t-1}) &= \frac{\text{count}(w_{t-1}w_{t})}{\sum_{w_{t'}}\text{count}(w_{t-1}w_{t'})}\\
               &= \texttt{count[prefix][wt] / totals[prefix]}
\end{align}

where $\texttt{prefix}$ is $w_{t-1}$ in this case.

```python
count = defaultdict(lambda: defaultdict(float))
total = defaultdict(float)

n = 2
for sequence in data:
    for i in range(len(sequence)-n+1):         # for each ngram
        ngram = tuple(sequence[i:i+n])
        prefix, word = ngram[:-1], ngram[-1]
        count[prefix][word] += 1               # count(w_{t-n+1}...w_t)
        total[prefix] += 1                     # count(w_{t-n+1}...w_{t-1})
```

Let's see if the counts and totals make sense:

- How many times did (A, B) occur? What about (B, B)?
- How many times did (A) occur? What about (C)?

```python
print("Counts:")
pprint(dict(count))
print("\nTotals:")
pprint(dict(total))
```

    Counts:
    {('<bos>',): defaultdict(<class 'float'>, {'A': 5.0}),
     ('A',): defaultdict(<class 'float'>, {'A': 8.0, 'B': 3.0, '<eos>': 2.0}),
     ('B',): defaultdict(<class 'float'>, {'B': 1.0, '<eos>': 2.0, 'C': 1.0}),
     ('C',): defaultdict(<class 'float'>, {'<eos>': 1.0})}

    Totals:
    {('<bos>',): 5.0, ('A',): 13.0, ('B',): 4.0, ('C',): 1.0}

#### Conditional probability queries

We can now query a conditional probability:

\begin{align}
\texttt{p(word|prefix)} =&\ \texttt{count[prefix][word] / totals[prefix]}
\end{align}

```python
queries = [('<bos>', 'A'),
           ('B', 'C')]

for query in queries:
    prefix, word = query[:-1], query[-1]
    p = count[prefix][word] / total[prefix]  # We'll discuss the case when `total[prefix] = 0` below.
    print("p( %s | %s) = \t%.5f" % (word, ', '.join(prefix), p))
```

    p( A | <bos>) = 	1.00000
    p( C | B) = 	0.25000

**Exercise**: Look at the training set and convince yourself that these conditional probabilities are correct according to the count-based estimation procedure.

#### Sequence Probability

We can compute the probability of a sequence using the conditional probabilities along with the chain rule of probability:

\begin{align}
p(w_1,\ldots,w_T)&\approx\prod_{t=1}^T p(w_t|w_{t-1})
\end{align}

(Here $w_0$ is `<bos>` and $w_T$ is `<eos>`)

```python
sequence = ['<bos>', 'A', 'A', 'B', '<eos>']

def sequence_p(sequence, log=False):
    total_p = 1

    for i in range(len(sequence)-n+1):
        ngram = tuple(sequence[i:i+n])
        prefix = ngram[:-1]
        word = ngram[-1]
        p = count[prefix][word] / max(total[prefix], 1)
        if log:
            print("p(%s | %s) =\t%.3f" % (word, ', '.join(prefix), p))

        total_p *= p
    return total_p

print("\nProduct: p(%s) = %.3f" % (''.join(sequence[1:-1]), sequence_p(sequence, log=True)))
```

    p(A | <bos>) =	1.000
    p(A | A) =	0.615
    p(B | A) =	0.231
    p(<eos> | B) =	0.500

    Product: p(AAB) = 0.071

### Real Example: Dialogue Utterances

Now lets use the same ideas on a more realistic text corpus.

We will use utterances from a dialogue dataset called **Persona-Chat**. This dataset is relatively small and centers on a single domain, but it is simple and interpretable for our purposes here.

We'll also use an off-the-shelf ngram modeling package called `KenLM`.

#### Loading the dataset

```python
!ls
```

    data  img  kenlm  models  ngram_lms.ipynb

```python
train = []
valid = []
import jsonlines

for ds, name in [(train, 'train'), (valid, 'valid')]:
    for line in jsonlines.Reader(open('data/personachat/personachat_all_sentences_%s.jsonl' % name, 'r')):
        ds.append(line['tokens'])

vocab = list(set([t for ts in train for t in ts]))
print("Number of train examples: %d" % (len(train)))
print("Number of valid examples: %d" % (len(valid)))
print("Vocab size: %d" % (len(vocab)))

print("\nExamples:")
pprint(train[:3])
```

    Number of train examples: 133176
    Number of valid examples: 16181
    Vocab size: 19153

    Examples:
    [['i', 'am', 'doing', 'great', 'except', 'for', 'the', 'allergies', '.'],
     ['i', 'am', 'a', 'woman', 'what', 'about', 'you', '.'],
     ['i', 'thought', 'you', 'were', 'a', 'college', 'kid', '.']]

### KenLM

KenLM estimates n-gram language models using **modified Kneser-Ney smoothing**, and has a fast and memory-efficient implementation.
- While we won't go into details here, **smoothing** is a technique used to account for ngrams that do not occur in the training corpus.
- Normally, these ngrams would receive zero-probability mass. Smoothing ensures every ngram receives some probability.

Please see page 48 of the [lecture note](https://github.com/nyu-dl/NLP_DL_Lecture_Note/blob/master/lecture_note.pdf) for an overview of Kneser-Ney smoothing, and [[Chen & Goodman 1998]](https://dash.harvard.edu/bitstream/handle/1/25104739/tr-10-98.pdf?sequence=1) for further details.

```python
KENLM_DIR='./kenlm' # for colab and local computers
# KENLM_DIR='/home/jupyter/kenlm' # for GCP
```

#### Tokenize data

```python
if not os.path.exists('data/tokenized'):
    os.makedirs('data/tokenized')
with open('data/tokenized/pchat_train', 'w') as f:
    for line in tqdm(train):
        f.write(' '.join(line))
        f.write('\n')
```

    100%|██████████| 133176/133176 [00:00<00:00, 711179.93it/s]

#### Build kenlm n-gram models

This uses the `kenlm` commands `lmplz` to estimate the language model, then `build_binary` to convert it to an efficient format. We load the resulting model using the `kenlm` python wrapper.

We do this for n-gram models of order `2,3,4`:

```python
import kenlm

data_prefix = 'pchat'
dataset = 'pchat_train'
if not os.path.exists('models'):
    os.makedirs('models')

models = {}
for n in [2,3,4]:
    model_temp = 'models/%s_%d.arpa' % (data_prefix, n)
    model_name = 'models/%s_%d.klm' % (data_prefix, n)
    ! cat ./data/tokenized/$dataset | $KENLM_DIR/build/bin/lmplz -o $n > $model_temp
    ! $KENLM_DIR/build/bin/build_binary $model_temp $model_name
    models[n] = kenlm.LanguageModel(model_name)
```

    === 1/5 Counting and sorting n-grams ===
    File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
    Unigram tokens 1602042 types 19156
    === 2/5 Calculating and sorting adjusted counts ===
    Chain sizes: 1:229872 2:13355060224
    /home/aims/Desktop/kenlm/util/scoped.cc:20 in void* util::{anonymous}::InspectAddr(void*, std::size_t, const char*) threw MallocException because `!addr && requested'.
    Cannot allocate memory for 13355060224 bytes in malloc
    Aborted
    Reading models/pchat_2.arpa
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    End of file Byte: 0
    ERROR
    === 1/5 Counting and sorting n-grams ===
    File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
    Unigram tokens 1602042 types 19156
    === 2/5 Calculating and sorting adjusted counts ===
    Chain sizes: 1:229872 2:4645238272 3:8709822464
    Statistics:
    1 19156 D1=0.588857 D2=1.0348 D3+=1.3388
    2 231267 D1=0.722265 D2=1.09097 D3+=1.44737
    3 617624 D1=0.798924 D2=1.0703 D3+=1.33013
    Memory estimate for binary LM:
    type       kB
    probing 16763 assuming -p 1.5
    probing 18193 assuming -r models -p 1.5
    trie     6683 without quantization
    trie     3625 assuming -q 8 -b 8 quantization
    trie     6354 assuming -a 22 array pointer compression
    trie     3296 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
    === 3/5 Calculating and sorting initial probabilities ===
    Chain sizes: 1:229872 2:3700272 3:12352480
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 4/5 Calculating and writing order-interpolated probabilities ===
    Chain sizes: 1:229872 2:3700272 3:12352480
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 5/5 Writing ARPA model ===
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    Name:lmplz	VmPeak:13193612 kB	VmRSS:23260 kB	RSSMax:3049068 kB	user:1.2624	sys:0.816377	CPU:2.07881	real:1.97458
    Reading models/pchat_3.arpa
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    SUCCESS
    === 1/5 Counting and sorting n-grams ===
    File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
    Unigram tokens 1602042 types 19156
    === 2/5 Calculating and sorting adjusted counts ===
    Chain sizes: 1:229872 2:2273201664 3:4262253056 4:6819604992
    Statistics:
    1 19156 D1=0.588857 D2=1.0348 D3+=1.3388
    2 231267 D1=0.722265 D2=1.09097 D3+=1.44737
    3 617624 D1=0.821472 D2=1.14168 D3+=1.40217
    4 932153 D1=0.863174 D2=1.12888 D3+=1.29333
    Memory estimate for binary LM:
    type       kB
    probing 36767 assuming -p 1.5
    probing 41816 assuming -r models -p 1.5
    trie    15838 without quantization
    trie     8356 assuming -q 8 -b 8 quantization
    trie    14567 assuming -a 22 array pointer compression
    trie     7085 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
    === 3/5 Calculating and sorting initial probabilities ===
    Chain sizes: 1:229872 2:3700272 3:12352480 4:22371672
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 4/5 Calculating and writing order-interpolated probabilities ===
    Chain sizes: 1:229872 2:3700272 3:12352480 4:22371672
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 5/5 Writing ARPA model ===
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    Name:lmplz	VmPeak:13210000 kB	VmRSS:45052 kB	RSSMax:2658428 kB	user:1.80083	sys:0.928808	CPU:2.72967	real:2.42089
    Reading models/pchat_4.arpa
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    SUCCESS

## Evaluation

### Perplexity

Intuitively, a good model should assign high probabilities to sequences from the 'true' distribution that it is modeling.

A common way of quantifying this is with **perplexity**, a metric inversely-proportional to the probability that the model assigns to a set of sequences, e.g. a 'test set':

\begin{align}
\large \text{ppl}(p, D) &\large\ = 2^{-\frac{1}{N_{total}}\log_2 p(D)}
\end{align}

where $D=\{(w_1,\ldots,w_{N_i})_i\}_{i=1}^M$ is a dataset of $M$ sequences with total length $N_{\text{total}}=\sum_{i}N_i$.

Perplexity is defined on $[1,\infty)$, with 1 being a perfect model (assigning probability 1 to $D$), and a 'worse' model as perplexity increases.

Intuitively, _perplexity measures the average rank of the true next-token, when tokens are ordered by the model's conditional probabilities_.

#### Evaluate Perplexity

`kenlm` outputs log probabilities in **log base 10**. For the standard definition of perplexity we need **log base 2**. See the code below:

```python
def perplexity_kenlm(model, sequences):
    n_total = 0
    logp_total = 0
    for sequence in sequences:
        text = ' '.join(sequence)
        logp_total += model.score(text)
        n_total += len(sequence) + 1  # add 1 for </s>

    # Convert log10 to log2
    log2p_total = logp_total / np.log10(2)
    ppl = 2 ** (- (1.0 / n_total) * log2p_total)
    return ppl
```

```python
print("=== Train ===")
df = pd.DataFrame([(n, perplexity_kenlm(models[n], train)) for n in [2, 3, 4]], columns=['n', 'ppl'])
df.style.hide_index()
```

    === Train ===

        n	        ppl

                                2
                        39.9248

                                3
                        15.462

                                4
                        8.91961

```python
print("=== Valid ===")
df = pd.DataFrame([(n, perplexity_kenlm(models[n], valid)) for n in [2, 3, 4]], columns=['n', 'ppl'])
df.style.hide_index()
```

    === Valid ===

        n	        ppl

                                2
                        59.188

                                3
                        44.9771

                                4
                        43.2188

#### Sequence probabilities:
\begin{align}
p(w_1,\ldots,w_T)&\approx\prod_{t=1}^T p(w_t|w_{t-2},w_{t-1})\\
&=\sum_{t=1}^T \log p(w_t|w_{t-2},w_{t-1}).
\end{align}

where we use log probabilities in practice to avoid a product of many small numbers.

```python
sentences = [
    'i like my pet dog .',
    'i like my pet zebra .',
    'i like my pet lion .',
    'i live in the united states .',
    'i live in the united states of america .'
]

for sentence in sentences:
    print(sentence)
    for n in [2, 3, 4]:
        log10p = models[n].score(sentence)
        log2p = log10p / np.log10(2)
        print("n: %d\t logp: %.3f" % (n, log2p))
    print()
```

    i like my pet dog .
    n: 2	 logp: -29.216
    n: 3	 logp: -28.843
    n: 4	 logp: -29.219

    i like my pet zebra .
    n: 2	 logp: -32.303
    n: 3	 logp: -31.157
    n: 4	 logp: -32.101

    i like my pet lion .
    n: 2	 logp: -38.352
    n: 3	 logp: -41.017
    n: 4	 logp: -42.320

    i live in the united states .
    n: 2	 logp: -22.599
    n: 3	 logp: -20.095
    n: 4	 logp: -17.854

    i live in the united states of america .
    n: 2	 logp: -40.958
    n: 3	 logp: -26.627
    n: 4	 logp: -24.311


```python
%matplotlib inline
from IPython.display import display,Math
from sympy import *
init_session()
```

```python
import datetime
utime = 0
dt = datetime.datetime.fromtimestamp(utime,datetime.timezone.utc)
print("{0:d} に対応するUNIX時間は {1:s} です．".format(utime,str(dt)[:-6]))
```

```python
import time
now = time.time()
now_jst = str(datetime.datetime.fromtimestamp(now,datetime.timezone(datetime.timedelta(hours=9))))
now_utc = str(datetime.datetime.fromtimestamp(now,datetime.timezone.utc))
print("今は{0:s}です．\n(日本時間{1:s})\nUNIX時間は {2:d} になります．".format(now_utc,now_jst,int(now*10**3)))
```

```python
import time
import datetime
from IPython.display import HTML
from ipywidgets import interact
from ipywidgets import interact,Dropdown,IntSlider
@interact
def _(n=str(int(time.time()))):
    n = int(eval(n))
    dt = datetime.datetime.fromtimestamp(n,datetime.timezone.utc)
    return HTML("{0:d} に対応する時間は {1:s}".format(n,str(dt)[:-6]))

```

```python

```


[Home](home.ipynb) > Perception biases in networks with Homophily and Preferential Attachment

### *Notebooks for Social Network Analysis*

# Perception biases in networks with homophily and group sizes

Author: [Fariba Karimi](https://www.gesis.org/person/fariba.karimi)

Version: 0.1 (21.01.2020)

Please cite as: Karimi, Fariba (2020). Perception biases in networks with homophily and group sizes. *Notebooks for Social Network Analysis*. GESIS. url:xxx

<big>Significance</big>

People’s perceptions about the size of minority groups in social networks can be biased, often showing systematic over- or
underestimation. These social perception biases are often attributed to biased cognitive or motivational processes. Here we
show that both over- and underestimation of the size of a minority group can emerge solely from structural properties of social
networks. Using a generative network model, we show that these biases depend on the level of homophily, its asymmetric
nature and on the size of the minority group. Our model predictions correspond well with empirical data from a cross-cultural
survey and with numerical calculations from six real-world networks. We also identify circumstances under which individuals
can reduce their biases by relying on perceptions of their neighbours.

### For more information, please visit the following paper

## Introduction

People’s perceptions of their social worlds determine their own
personal aspirations1 and willingness to engage in different
behaviours, from voting and energy conservation to health
behaviour, drinking and smoking. Yet, when forming these perceptions,
people seldom have an opportunity to draw representative
samples from the overall social network, or from the general
population. Instead, their samples are constrained by the local
structure of their personal networks, which can bias their perception
of the relative frequency of different attributes in the general
population. For example, supporters of different candidates in the
2016 US presidential election formed relatively isolated Twitter
communities. Such insular communities can overestimate the relative
frequency of their own attributes in the overall society. This
has been documented in the literature on overestimation effects
including *false consensus*, looking-glass perception and, more generally,
social projection. In an apparent contradiction, it has also
been documented that people holding a particular view sometimes
underestimate the frequency of that view, as described in the literature
on *false uniqueness*, pluralistic ignorance and majority
illusion. These over- and underestimation errors, which we call
social perception biases, affect people’s judgements of minority- and
majority-group sizes.

In this study, we show empirically, analytically and numerically that a
simple network model can explain both over- and underestimation
in social perceptions, without assuming biased motivational or cognitive
processes.

At the individual level, we assume that individuals’ perceptions
are based on the frequency of an attribute in their personal networks
(their direct neighbourhoods). We define individual i’s social
perception bias as follows:

\begin{equation}
    B_{indv,i}=1/f_{m}\frac{\sum_{j\in\delta}{ x_j}}{k_i},
\end{equation}

where $f_m$ denotes the fraction of minority in the whole network, $\delta$ is the set of $i$'s neighbors, and $k_i$ is the degree of $i$. The group perdception bias is the average perception over all the members of the group.

Figure below describes the concept of the perception bias. The blue node $i$ in the center tends to underestimate the size of the minority nodes (orange nodes) in the homophilic networks and overestimate the size of the minority nodes in the heterophilic network:

**In this notebook**, I provide an intuitive walk through the construction of the perception biases in homophilic-preferential attachment network Model (BA-Homophily model). For constructing the BA-Homophily model, please check the other notebook.

## Dependencies and Settings

```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python

```

## Model

``homophilic_ba_graph(N,m,min_fraction,homophily)`` is the main function to generate the network. This function can be called from ``generate_homophilic_graph_symmetric.py``.

After generating the BA-Homophily model, we can measure the perception bias of each node by iterating over each node and calculating the perception bias. Finally, we average the perception bias across each group to calculate the group perception bias. The results are averaged over 20 independet simulations and stored at ``perception_bias/images/Fig2``.

### Draw the perception bias versus homophily and minority size

Here we plot perception bias of the minority and the majority group versus homophily and group sizes.

```python

plt.rcParams["font.family"] = "sans-serif"

fminor_list = [0.1, 0.2, 0.3, 0.4, 0.5]
fig = plt.figure(figsize = (10,6))
ha = [i/10. for i in range(11)]
hb = [1.-temp_h for temp_h in ha]
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)
ax1.axvspan(0.5,1.0,color='lightgrey', edgecolor = 'lightgrey', alpha=0.5)
ax2.axvspan(0.0,0.5, color='lightgrey', edgecolor = 'lightgrey', alpha=0.5)
cmap = ['Goldenrod', 'coral', 'Pink', 'Firebrick', 'Red']
for fi in range(len(fminor_list)) :
    fminor = fminor_list[fi]

    ''' Initial data'''
    Cdata = np.loadtxt('perception_bias/images/Fig2/ctest_fa%s.txt' %(fminor))
    fa = fminor
    fb = 1.-fminor

    minor_data = np.loadtxt('perception_bias/images/Fig2/neterr_minor_fa%s.txt' %(fminor))
    major_data = np.loadtxt('perception_bias/images/Fig2/neterr_major_fa%s.txt' %(fminor))
    c = Cdata[:,1]

    ''' Fig1(a) for the minority'''
    minorh = minor_data[:,0]
    minor_neterr = minor_data[:,1]
    minor_analytic_y = [2*((fa*ha[temp_c])/(ha[temp_c]*c[temp_c]+hb[temp_c]*(2.-c[temp_c])) )/fa for temp_c in range(len(c))]
    norm_minor_neterr = [(merr+fminor)/fminor for merr in minor_neterr]
    ax1.axhline(y=1, color = 'grey', ls = 'dashed')
    ax1.scatter(minorh, norm_minor_neterr, marker = 'o', edgecolor = 'black', s= 60,label = '$f_M=%s$' %fminor, facecolor = cmap[fi], zorder = 2)
    ax1.plot(minorh, minor_analytic_y, label = '',color = cmap[fi])
    ax1.set_xlabel(r'$h$', fontsize = 25)
    ax1.set_ylabel(r'$P_{group}$', fontsize = 25,labelpad = -5)
    ax1.xaxis.set_tick_params(labelsize=20)
    ax1.yaxis.set_tick_params(labelsize=20)
    ax1.set_xlim([0.0,1.0])
    ax1.set_ylim([0.0,10.0])

    ''' Fig1(b) for the majority'''
    majorh = major_data[:,0]
    major_neterr = major_data[:,1]
    major_analytic_y = [((fa*hb[temp_c])/(ha[temp_c]*c[temp_c]+hb[temp_c]*(2.-c[temp_c])) + ((c[temp_c]/(2.-c[temp_c]))*(fb*hb[temp_c])/(hb[temp_c]*c[temp_c]+ha[temp_c]*(2.-c[temp_c]))))/ (fa) for temp_c in range(len(c))]
    norm_major_neterr = [(mjerr+fminor)/fminor for mjerr in major_neterr]
    ax2.axhline(y=1, color = 'grey', ls = 'dashed')
    ax2.scatter(majorh, norm_major_neterr, marker = 'o', s = 60, edgecolor = 'black', label = '$f_M=%s$' %fminor, color = cmap[fi], zorder = 2)
    ax2.plot(majorh, major_analytic_y, label = '',color = cmap[fi])
    ax2.set_xlabel(r'$h$', fontsize = 25)
    ax2.xaxis.set_tick_params(labelsize=20)
    ax2.yaxis.set_tick_params(labelsize=20)
    ax2.set_xlim(0.0,1.0)
    ax2.set_ylim(0.0,10.0)

''' Set labels '''
ax1.text(0.3,10.3, '(a) Minority',fontsize = 20)
ax2.text(0.3,10.3,'(b) Majority', fontsize = 20)

ax1.text(0.57,0.4, 'Filter Bubble',fontsize = 14,style='italic')
ax2.text(0.04,0.4, 'Majority Illusion',fontsize = 14,style='italic')

ax1.legend(scatterpoints=1, fontsize =17,bbox_to_anchor=(2.2, 1.23), fancybox=True, ncol=5, handletextpad=0.1,columnspacing=0.3)

#fig.savefig('Fig2.pdf', bbox_inches = 'tight')
plt.show()

```

### Reducing the perception bias by asking the neighbors

To what extent and under what structural conditions can individuals reduce their perception
bias? To address this question, we consider perception biases of
individuals and their neighbours. We aggregated each individual’s
own perception of frequency of different attributes (ego) with the
averaged perceptions of the individual’s neighbours. This is similar to *DeGroot’s weighted belief formalization*.

#### Degree growth of the minority and the majority (comparing analytical and numerical analysis)

```python
import numpy as np
import matplotlib.pyplot as plt
import os, sys, inspect
import math

plt.rcParams["font.family"] = "sans-serif"

fminor_list = [0.2]

fig = plt.figure(figsize = (10,6))
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)

cmap = [ 'violet','yellowgreen']
for fi in range(len(fminor_list)) :

    fminor = fminor_list[fi]
    fa = fminor
    fb = 1.-fminor

    minor_data = np.loadtxt('perception_bias/images/Fig5/0hop_neterr_minor_fa%s.txt' %(fminor))
    major_data = np.loadtxt('perception_bias/images/Fig5/0hop_neterr_major_fa%s.txt' %(fminor))

    minorh = minor_data[:,0]
    minor_neterr = minor_data[:,1]
    minor_neterr_std = minor_data[:,2]

    hop_minor_data = np.loadtxt('perception_bias/images/Fig5/1.0_hop_neterr_minor_fa%s_avg_sum.txt' %(fminor))
    hop_major_data = np.loadtxt('perception_bias/images/Fig5/1.0_hop_neterr_major_fa%s_avg_sum.txt' %(fminor))

    hop_y_min_mean = hop_minor_data[:,1]
    hop_y_min_std = hop_minor_data[:,2]

    '''Fig.5(a) for the minority '''
    ax1.axhline(y= 1, color = 'grey', ls = 'dashed')
    ax1.plot(minorh, (minor_neterr)/fminor, marker = 'o', label = 'ego', color = cmap[0] , mec = 'black')
    ax1.errorbar(minorh, (minor_neterr)/fminor, yerr = minor_neterr_std,  color = cmap[0] )
    ax1.plot(minorh, (hop_y_min_mean)/fminor  , marker = '^', label = '1-hop weighted avg.' , color = cmap[1] , mec = 'black')
    ax1.errorbar(minorh, (hop_y_min_mean)/fminor, yerr= hop_y_min_std  ,  color = cmap[1] )
    ax1.xaxis.set_tick_params(labelsize=20)
    ax1.yaxis.set_tick_params(labelsize=20)
    ax1.set_xlabel(r'$h$', fontsize = 24)
    ax1.set_ylabel(r'$P_{indv.}$', fontsize = 24)
    ax1.legend(numpoints = 1, loc = 'best' ,fontsize = 20)

    '''Fig.5(b) for the majority'''
    hop_y_maj_mean = hop_major_data[:,1]
    hop_y_maj_std = hop_major_data[:,2]
    majorh = major_data[:,0]
    major_neterr = major_data[:,1]
    major_neterr_std = major_data[:,2]
    ax2.axhline(y= 1, color = 'grey', ls = 'dashed')
    ax2.plot(majorh, (major_neterr)/fminor , marker = 'o', label = 'ego' , color = cmap[0] , mec = 'black')
    ax2.errorbar(minorh, (major_neterr)/fminor, yerr = major_neterr_std,  color = cmap[0] )
    ax2.plot(majorh, (hop_y_maj_mean)/fminor , marker = '^', label = '1-hop weighted avg.' , color = cmap[1] , mec = 'black')
    ax2.errorbar(minorh, (hop_y_maj_mean)/fminor, yerr = hop_y_maj_std,  color = cmap[1] )
    ax2.set_xlabel(r'$h$', fontsize = 24)
    ax2.xaxis.set_tick_params(labelsize=20)
    ax2.yaxis.set_tick_params(labelsize=20)

ax1.xaxis.set_ticks(np.arange(0, 1.2, 0.2))
ax2.xaxis.set_ticks(np.arange(0, 1.2, 0.2))

ax1.text(0.28,5.1, '(a) Minority',fontsize = 20)
ax2.text(0.28,5.1, '(b) Majority',fontsize = 20)

ax1.legend(numpoints=1, fontsize =17,bbox_to_anchor=(1.6, 1.2), fancybox=True, ncol=5, handletextpad=0.1,columnspacing=0.3)

plt.show()
#plt.savefig('Fig5.pdf', bbox_inches = 'tight')

```

## Acknowledgments

The code for plotting perception biases are written by Dr. [Eun Lee](https://www.eunlee.net/).

## Literature
Lee, E., Karimi, F., Wagner, C., Jo, H. H., Strohmaier, M., & Galesic, M. (2019). Homophily and minority-group size explain perception biases in social networks. Nature human behaviour, 3(10), 1078-1087.

Karimi, F., Génois, M., Wagner, C., Singer, P., & Strohmaier, M. (2018). Homophily influences ranking of minorities in social networks. Scientific reports, 8(1), 1-12.(https://www.nature.com/articles/s41598-018-29405-7)

DeGroot, M. H. (1974). Reaching a consensus. Journal of the American Statistical Association, 69(345), 118-121.

Centola, D. (2011). An experimental study of homophily in the adoption of health behavior. Science, 334(6060), 1269-1272.

Galesic, M., de Bruin, W. B., Dumas, M., Kapteyn, A., Darling, J. E., & Meijer, E. (2018). Asking about social circles improves election predictions. Nature Human Behaviour, 2(3), 187-193.