```python
%matplotlib inline

import matplotlib
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["font.size"] = 18
```

# Radiation Interatctions with Matter

### Learning Objectives

- Define uncollided flux
- Define linear interaction coefficient
- Apply linear interaction coefficients to a slab problem
- Identify the units of intensity, flux density, fluence, reaction rate
- Compare linear interaction coefficient and cross section
- Calculate uncollided flux in a medium
- Calculate mean free path of a particle in a medium
- Define the half thickness in a medium
- Apply the concept of buildup factor to attenuation in a slab
- Define microscopic cross section
- Calculate macroscopic cross sections, given a microscopic cross section
- Calculate the mass interaction coefficients of mixtures
- Calculate flux density
- Calculate Reaction Rate Density
- Recognize the dependence of flux on energy, position, and time
- Define radiation fluence
- Calculate uncollided flux density from isotropic point sources
- Apply the Kelin-Nishina formula to Compton Scattering
- Compare energy dependence of photon interaction cross sections
- Describe energy dependence of neutron interaction cross sections
- Recognize the comparative range of heavy vs. light particles
- Recognize the comparative range of charged particles

## Linear Interaction Coefficient

- The interaction of radiation with matter is always statistical in nature, and, therefore, must be described in probabilistic terms.

Consider a particle travelling through a homogeneous material.

\begin{align}
P_i(\Delta x) &= \mbox{probability the particle, causes a reaction of type i in distance }\Delta x\\
\end{align}

Empirically, we find that this probability becomes constant as $\Delta x \longrightarrow 0$. Thus:

\begin{align}
\mu_i &= \lim_{\Delta x \rightarrow 0}\frac{P_i(\Delta x)}{\Delta x}\\
\end{align}

Facts about $\mu_𝑖$:

- $\mu_i$ is an *intrinsic* property of the material for a given incident particle and interaction.
- $\mu_i$ is independent of the path length traveled prior to the interaction.
- $\mu_i$ may represent many types of interaction (scattering: $\mu_s$, absorption: $\mu_a$, ...)
- $\mu_i$ typically depends on particle energy

The probability, per unit path length, that a neutral particle undergoes some sort of reaction, is the sum of the probabilities, per unit path length of travel, for each type :

\begin{align}
\mu_t(E) = \sum_i \mu_i(E)
\end{align}

## Think Pair Share:

What are the units of the linear interaction coefficient?

### Attenuation of Uncollided Flux

Imagine a plane of neutral particles strike a slab of some material, normal to the surface.

We can describe this using $\mu_t$ or, equivalently, the macroscopic total cross section $\Sigma_t$.

\begin{align}
I(x) &= I_0e^{-\mu_t x}\\
I(x) &= I_0e^{-\Sigma_t x}\\
\end{align}

where

\begin{align}
        I(x) &= \mbox{uncollided intensity at distance x}\\
        I_0 &= \mbox{initial uncollided intensity}\\
        \mu_t &= \mbox{total linear interaction coefficient} \\
        \Sigma_t &= \mbox{macroscopic total cross section} \\
        x &= \mbox{distance into material [m]}\\
\end{align}

```python
import math
def attenuation(distance, initial=100, sig_t=1):
    """This function describes neutron attenuation into the slab"""
    return initial*math.exp(-sig_t*distance)

```

Rather than intensity, one can find the probability density:

We have a strong analogy between decay and attenuation, as above. In the case of decay the probability of decay in a time interval dt is:

\begin{align}
P(t)dt &= \lambda e^{-\lambda t}dt\\
 &= \mbox{probability of decay in interval dt}
\end{align}

From this, one can find the mean lifetime of a neutron before decay:

\begin{align}
\bar{t} &= \int_0^\infty t'P(t')dt'\\
        &= \int_0^\infty t'\lambda e^{-\lambda t'}dt'\\
        &= \frac{1}{\lambda}
\end{align}

In the case of attenuation:
\begin{align}
P(x)dx &= \Sigma_te^{-\Sigma_tx}dx
\end{align}

Such that:

\begin{align}
P(x)dx &= \mu_t e^{-\mu_t x}dx\\
 &= \Sigma_t e^{-\Sigma_t x}dx\\
 &= \mbox{probability of interaction in interval dx}
\end{align}

So, the mean free path is:

\begin{align}
\bar{l} &= \int_0^\infty x'P(x')dx'\\
        &= \int_0^\infty x'\Sigma_te^{-\Sigma_t x'}dx'\\
        &= \frac{1}{\Sigma_t}
\end{align}

Or, equivalently in $\mu_t$ notation:

\begin{align}
\bar{x} &= \int_0^\infty x'P(x')dx'\\
        &= \int_0^\infty x'\mu_te^{-\mu_t x'}dx'\\
        &= \frac{1}{\mu_t}
\end{align}

```python
def prob_dens(distance, initial=100, sig_t=1):
    return sig_t*attenuation(distance, initial=100, sig_t=1)

```

```python
sig_t = 0.2
i_0 = 100

# This code plots attenuation
import numpy as np
z = np.arange(24)
y = np.arange(24)
x = np.arange(24)
for h in range(0,24):
    x[h] = h
    y[h] = attenuation(h, initial=i_0, sig_t=sig_t)
    z[h] = prob_dens(h, initial=i_0, sig_t=sig_t)

# creates a figure and axes with matplotlib
fig, ax = plt.subplots()
scatter = plt.scatter(x, y, color='blue', s=y*20, alpha=0.4)
ax.plot(x, y, color='red')
ax.plot(x, z, color='green')

# adds labels to the plot
ax.set_ylabel('Percent of Neutrons')
ax.set_xlabel('Distance into slab')
ax.set_title('Attenuation')

# adds tooltips
import mpld3
labels = ['{0}% intensity'.format(i) for i in y]
tooltip = mpld3.plugins.PointLabelTooltip(scatter, labels=labels)
mpld3.plugins.connect(fig, tooltip)

mpld3.display()
```

## Half-thickness

In another analog to decay, the **half-thickness** of a material is the distance required for half of the incident radiation to interact with a medium:

\begin{align}
\frac{I(x_{1/2})}{I(0)} &= e^{-\mu_t x_{1/2}}\\
\implies x_{1/2} &= \frac{\ln{2}}{\mu_t}
\end{align}

## Think pair share:
What is the concept in the context of decay that is analogous to the half-thickness?

## Microscopic Cross Sections

- The microscopic cross section $\sigma_i$ is  the likelihood of the event per unit area.
- The macroscopic cross section $\Sigma_i$ is  the likelihood of the event per unit area of a certain density of target isotopes.
- The macroscopid cross section $\Sigma_i$ is equivalent to the linear interaction coefficient $\mu_i$, but we tend to use $\Sigma_i$ in nuclear interactions, reserving $\mu_i$ for photon interactions.

\begin{align}
\mu_i &= \mbox{linear interaction coefficient}\\
\Sigma_i &= \mbox{macroscopic cross section}\\\\
 &= \sigma_i N\\
 &= \sigma_i \frac{\rho N_a}{A}\\
 \mbox{where }& \\
 N &= \mbox{atom density of medium}\\
 \rho &= \mbox{mass density of the medium}\\
 N_a &= \mbox{Avogadro's number}\\
 A &= \mbox{atomic weight of the medium}
\end{align}

```python
def macroscopic_xs(micro, N):
    """Returns the macroscopic cross section [cm^2] or [barns]

    Parameters
    ----------
    micro: double
        microscopic cross section [cm^2] or [barns]
    N: double
        atom density in the medium [atoms/cm^3]
    """
    return micro*N
```

```python
def NA():
    """Returns Avogadro's number
    6.022x10^23 atoms per mole
    """
    return 6.022E23

def num_dens_from_rho(rho, na, a):
    """The atomic number density.
    That is, the concentration of atoms or molecules per unit volume (V)

    Parameters
    -----------
    rho : double
        material density (in units like g/cm^3 or kg/m^3) of the sample
    na : double
        Avogadro's number
    a : double
        The atomic or molecular weight of the atom or molecule of interest
    """
    return rho*na/a
```

## Example:
Imagine a beam of neutrons striking a body of water, $H_2O$. Many will be absorbed by the hydrogen in the water, particularly $^1H$.

```python
# Find the macroscpic absorption cross section
# of the 1H in H2O
sig_1h = 0.333 # barns

# First, molecular density of water
rho_h2o = 1 # g/cm^3
a_h2o = 18.0153 # g/mol
n_h2o = num_dens_from_rho(rho_h2o, NA(), a_h2o) # molecules water / cm^3
n_h2o_barn = n_h2o/10**(24) # 10^24 molecules water / cm^3
print('n_h2o [1/cm^3] = ', n_h2o)
print('n_h2o [10^(24)/cm^3] = ', n_h2o_barn)

# Now, there are two Hydrogens in each molecule of water, so:
macroscopic_h1 = macroscopic_xs(sig_1h, 2*n_h2o_barn)
print('absorption in water from 1H = ', macroscopic_h1)
```

    n_h2o [1/cm^3] =  3.342714248444378e+22
    n_h2o [10^(24)/cm^3] =  0.033427142484443784
    absorption in water from 1H =  0.02226247689463956

### Mixtures
In a medium that is a mixture of isotopes (e.g. $H_2O$), we can calculate the total macroscopic cross section based on individual microscopic cross sections and number densities for each component of the mixture. We may need to include information about relative isotopic abundances (f).

For the same problem as above (neutrons striking a body of water) we can calculate the absorption by *all* isotopes in the $H_2O$.

\begin{align}
\mu^{H_2O} \equiv \Sigma^{H_2O} &= N^1\sigma_a^1 + N^2\sigma_a^2 + N^{16}\sigma_a^{16}
+ N^{17}\sigma_a^{17}  + N^{18}\sigma_a^{18}\\
&= f^1N^H\sigma_a^1 + f^2N^H\sigma_a^2 + f^{16}N^O\sigma_a^{16} + f^{17}N^O\sigma_a^{17} + f^{18}N^O\sigma_a^{18}
\end{align}

Superscripts 1, 2, 16, 17, and 18 indicate isotopes $^1H$, $^2H$, $^{16}O$,$^{17}O$, and $^{18}O$.

\begin{align}
N^H = 2N^{H_2O}\\
N^{O} = N^{H_2O}\\
N^{H_2O} = \frac{\rho^{H_2O}N_a}{A^{H_2O}}
\end{align}

Thus:
\begin{align}
\mu^{H_2O} \equiv \Sigma^{H_2O} &= N^{H_2O}\left[2f^1\sigma_a^1 + 2f^2\sigma_a^2 + f^{16}\sigma_a^{16} + f^{17}\sigma_a^{17} + f^{18}\sigma_a^{18}\right]
\end{align}

```python
# We need a lot of data

# Abundances
f_1 = 0.99985
f_2 = 0.00015
f_16 = 0.99756
f_17 = 0.00039
f_18 = 0.00205

# Then, microscopic absorption cross sections
sig_1 = 0.333
sig_2 = 0.000506
sig_16 = 0.000190
sig_17 = 0.239
sig_18 = 0.000160

macroscopic_h2o = n_h2o_barn*(2*f_1*sig_1
                              + 2*f_2*sig_2
                              + f_16*sig_16
                              + f_17*sig_17
                              + f_18*sig_18)
print('absorption in water from all isos = ', macroscopic_h2o,"\n",
     'while absorption in water from 1H = ', macroscopic_h1,"\n",
     'Thus, absorption in water is mostly from 1H.')
```

    absorption in water from all isos =  0.02226860496564809
     while absorption in water from 1H =  0.02226247689463956
     Thus, absorption in water is mostly from 1H.

### Reaction Rates

- The microscopic cross section is just the likelihood of the event per unit area.
- The macroscopic cross section is just the likelihood of the event per unit area of a certain density of target isotopes.
- The reaction rate is the macroscopic cross section times the flux of incident neutrons.

\begin{align}
R_{i,j}(\vec{r}) &= N_j(\vec{r})\int dE \phi(\vec{r},E)\sigma_{i,j}(E)\\
R_{i,j}(\vec{r}) &= \mbox{reactions of type i involving isotope j } [reactions/cm^3s]\\
N_j(\vec{r}) &= \mbox{number of nuclei participating in the reactions } [\#/cm^3]\\
E &= \mbox{energy} [MeV]\\
\phi(\vec{r},E)&= \mbox{flux of neutrons with energy E at position i } [\#/cm^2s]\\
\sigma_{i,j}(E)&= \mbox{cross section } [cm^2]\\
\end{align}

This can be written more simply as $R_x = \Sigma_x I N$, where I is intensity of the neutron flux.

Using flux notation, the density of ith type of neutron interaction with isotope j, per unit time is:

\begin{align}
R_{i,j}(\vec{r}) = \Sigma_{i,j}\phi(\vec{r})
\end{align}

### Reaction Rate Example: Fission Source term

An example of an important use of reaction rates is the source of neutrons in a reactor are the neutrons from fission.

\begin{align}
s &=\nu \Sigma_f \phi
\end{align}

where

\begin{align}
s &= \mbox{neutrons available for next generation of fissions}\\
\nu &= \mbox{the number born per fission}\\
\Sigma_f &= \mbox{the number of fissions in the material}\\
\phi &= \mbox{initial neutron flux}
\end{align}

This can also be written as:

\begin{align}
s =& \nu\Sigma_f\phi\\
  =& \nu\frac{\Sigma_f}{\Sigma_{a,fuel}}\frac{\Sigma_{a,fuel}}{\Sigma_a}{\Sigma_a} \phi\\
  =& \eta f {\Sigma_a} \phi\\
\eta =& \frac{\nu\Sigma_f}{\Sigma_{a,fuel}} \\
      =& \mbox{number of neutrons produced }\\
      & \mbox{  per neutron absorbed by the fuel}\\
      =& \mbox{"neutron reproduction factor"}\\
f =& \frac{\Sigma_{a,fuel}}{\Sigma_a} \\
   =& \mbox{number of neutrons absorbed in the fuel}\\
   &\mbox{  per neutron absorbed anywhere}\\
   =&\mbox{"fuel utilization factor"}\\
\end{align}

This absorption and flux term at the end seeks to capture the fact that some of the neutrons escape. However, if we assume an infinite reactor, we know that all the neutrons are eventually absorbed in either the fuel or the coolant, so we can normalize by $\Sigma_a\phi$ and therefore:

\begin{align}
k_\infty &= \frac{\eta f \Sigma_a\phi}{\Sigma_a \phi}\\
&= \eta f
\end{align}

## Flux density from Point Source
Finding $\phi(\vec{r}0$ generally requires *particle transport calculations.*

However, in some simple practical situations, the flux density can be approximated by the flux density of uncollided source particles.

### Point Source in Vacuum

Consider a source of particles:

- it emits $S_p$ particles per unit time
- all particles have energy E
- and they are emitted radially outward into an infinite vacuum
- isotropically (equally in all directions)
- from a single point in space

### Think-pair share:

- How many interactions occur?

### At a radius r:
Because the source is isotropic, each unit area on an imaginary spherical shell of radius $r$ has the same number of particles crossing it. Thus:

\begin{align}
\phi^o(r) &= \mbox{uncollided flux at radius r in any direction}\\
&= \frac{S_p}{4\pi r^2}
\end{align}

```python
def phi_o_r(r, s):
    """Returns the uncolided flux at radius r
    due to an isotropic point source in a vacuum

    Parameters
    -----------
    r : double
        radius away from the point [length]
    s : double
        point source strength [particles/time]
    """
    return s/(4*math.pi*pow(r,2))
```

```python
s=200

plt.plot(range(1,10), [phi_o_r(r, s) for r in range(1,10)])
```

The plot above, this $1/r^2$ reduction in flux and reaction rate, is occaisionally called "geometric attenuation".

```python
# The below IFrame displays Page 189 of your textbook:
# Shultis, J. K. (2016). Fundamentals of Nuclear Science and Engineering Third Edition,
# 3rd Edition. [Vitalsource]. Retrieved from https://bookshelf.vitalsource.com/#/books/9781498769303/
# Please take note of Figure 7.2

from IPython.display import IFrame
IFrame("https://bookshelf.vitalsource.com/books/9781498769303/pageid/211", width=1000, height=500)

```

## Point Source in an Attenuating Medium
So, the unollided flux is
\begin{align}
\phi^o(r) &= \frac{S_p}{4\pi r^2}
\end{align}

### A small volume

At a distance r, we place a homogeneous mass with a volume $\Delta V_d$. The interaction rate $R_d$ in the mass is:

\begin{align}
&R^o(r)=\mu_d(E)\Delta V_d\frac{S_p}{4\pi r^2}\\
\mbox{where}&\\
&\mu_d(E)=\mbox{linear interaction coefficient in the volume}
\end{align}

### An inifinite volume

From this, we can imagine the point source embeeded in an infinite medium of this material. A detector is at distance r in the volume:

\begin{align}
&\phi^o(r) = \frac{S_p}{4\pi r^2}e^{-\mu r}\\
\mbox{where}&\\
&e^{-\mu r}=\mbox{material attenuation}
\end{align}

### A slab shield

Imagine a slab shield, thickness t, at a distance r, between the point source and a detector.

\begin{align}
&\phi^o(r) = \frac{S_p}{4\pi r^2}e^{-\mu t}\\
\mbox{where}&\\
&t=\mbox{thickness of the slab}
\end{align}

If it were made of a series of materials $i$, with coefficients $\mu_i$, and thicknesses $t_i$:

\begin{align}
&\phi^o(r) = \frac{S_p}{4\pi r^2}e^{\sum_i -\mu_i t_i}\\
\mbox{where}&\\
&\mu_i=\mbox{linear interaction coefficient of ith slab}\\
&t_i=\mbox{thickness of ith slab}
\end{align}

### Heterogeneous Medium

An arbitrary heterogeneous medium can be described as having an interaction coefficient $\mu(\vec{r})$ at any point $\vec{r}$ in the medium, a funciton of position in the medium.

\begin{align}
&\phi^o(r) = \frac{S_p}{4\pi r^2}e^{\left[-\int_0^r \mu(s) ds\right]}\\
\end{align}

## Polyenergetic Point Source

- Previous examples assume a **monoenergetic** point source (particles of a single energy, E).
- But, a single source can emit particles at several discrete energies, or even a continuum of energies.

Let's define some variables:

\begin{align}
f_i &= \mbox{fraction of the source emitted with energy }E_i\\
E_i &= \mbox{discrete energy of }f_iS_p\mbox{ particles}\\
S_p &= \mbox{still the number of particles emitted from the point source}
\end{align}

The total interaction rate caused by uncollided particles streaming through a small volume mass at distance r from the source is the following, **for some set of i discrete energies**.

\begin{align}
R^o(r)=\sum_i\frac{S_p f_i\mu_d(E_i) \Delta V_d}{4\pi r^2}e^{\left[-\int_0^r \mu(s,E_i) ds\right]}\\
\end{align}

If the source emits a continuum of energies, it's best to define the fraction $f_i$ as a differential probability:

\begin{align}
N(E)dE\mbox{the probability that a source particle is emitted with energy in dE about E}
\end{align}

With this definition, the sum over discrete energies becomes an integral.

\begin{align}
R^o(r)=\int_o^\infty \left[\frac{S_p N(E)\mu_d(E) \Delta V_d}{4\pi r^2}e^{\left[-\int_0^r \mu(s,E) ds\right]}\right]dE\\
\end{align}

Please note, you may see many nuclear texts list the dE first in the integral... don't be bamboozled. This is equivalent to the above:

\begin{align}
R^o(r)=\int_o^\infty dE\frac{S_p N(E)\mu_d(E) \Delta V_d}{4\pi r^2}e^{\left[-\int_0^r \mu(s,E) ds\right]}
\end{align}

### Example 7.4 from your book (Shultis & Faw)

A point source with an activity of 500 Ci emits 2-MeV photons with a frequency of 70% per decay.

\begin{align}
S_p = 500 Ci\\
f_2 = 0.7\\
\end{align}

What is the flux density of 2-MeV photons 1 meter from the source?

```python
s_p = 500 # Ci
f_2 = 0.7 # fraction emitted at 2MeV
mu = 1.0/187.0 # mean free path of 2MeV photon in air is 187m

# first, convert S_p is in number of particles per decay (Bq)
bq_to_ci = 3.7e10 # Bq/Ci
s_p = s_p*bq_to_ci

# Now, find uncollided flux of 2MeV photons at 1 m
r = 1.0 #m
s = s_p*f_2 # just want 2MeV photons
phi = phi_o_r(r, s)
print("Uncollided flux is : ", phi)

# Uh oh, we forgot the material attenuation!
phi = phi_o_r(r, s)*math.exp(-mu*r)
print("Uncollided flux with attenuation is : ", phi)
```

    Uncollided flux is :  1030528256520.0223
    Uncollided flux with attenuation is :  1025032118881.2917

### Think Pair Share

What are the units of $\phi^o$, above?

# Photon Interactions

**Recall:**

\begin{align}
c &= \mbox{speed of light}\\
  &=2.9979\times10^8\left[\frac{m}{s}\right]\\
E &= \mbox{photon energy}\\
  &=h\nu\\
  &=\frac{hc}{\lambda}\\
h &= \mbox{Planck's constant}\\
  &= 6.62608\times10^{−34} [J\cdot s] \\
\nu &=\mbox{photon frequency}\\
\lambda &= \mbox{photon wavelength}
\end{align}

**Nota bene:**
- **10eV - 20MeV** photons are important in radiation sheilding
- At **10eV - 20MeV**, only photoelectric effect, pair production, and Compton Scattering are significant

Figure from: "Radiation Interactions with Tissue." Radiology Key. Jan 8 2016.

Figure from: Cullen, D. E. 1994. "Photon and Electron Interaction Databases and Their Use in Medical Applications." UCRL-JC--117419. Lawrence Livermore National Lab. http://inis.iaea.org/Search/search.aspx?orig_q=RN:26035330.

## Klein Nishina

The total Compton cross section, per atom with Z electrons, based on the free-electron approximation, is given by the well-known Klein-Nishina formula [Evans 1955]:

\begin{align}
\sigma_c(E) =\pi Zr_e^2\lambda\left[(-2\lambda - 2\lambda^2)\ln{\left(1+\frac{2}{\lambda}\right)} + \frac{2(1+9\lambda + 8\lambda^2 + 2\lambda^3)}{(\lambda + 2)^2}\right]
\end{align}

Here $\lambda \equiv \frac{m_ec^2}{E}$, a dimensionless quantity, and $r_e$ is the classical electron radius. The value of $r_e$ is given by:

\begin{align}
r_e &\equiv \frac{e^2}{4\pi\epsilon_om_ec^2}\\
&= 2.8179\times10^{-13}cm
\end{align}

### Think pair share:
Conceptually, in the above equation:

- what is $r_e$?
- what is $e$?
- what is $\epsilon_o$?
- what is $m_ec^2$?

### Total Photon Cross Section
Various types of incoherent scattering, including Compton, are actually present in that intermediate energy range. It is occaisionally important to correct for all types of incoherent scattering, but it can typically be assumed to be primarily Compton scattering.

For photons, then $\mu$ becomes:

\begin{align}
\mu(E)&\equiv N\left[\sigma_{ph}(E) + \sigma_{inc}(E) + \sigma_{pp}(E)\right]\\
      &\simeq N\left[\sigma_{ph}(E) + \sigma_{c}(E) + \sigma_{pp}(E)\right]\\
    N &= \mbox{atom density}\\
      &= \frac{\rho N_a}{A}
\end{align}

It is common to denote this as the total mass interaction coefficient:

\begin{align}
\frac{\mu}{\rho} &= \frac{N_a}{A}\left[\sigma_{ph}(E) + \sigma_{c}(E) + \sigma_{pp}(E)\right]\\
&= \frac{N_a}{A}\left[\frac{\mu_{ph}(E)}{\rho} + \frac{\mu_{c}(E)}{\rho} + \frac{\mu_{pp}(E)}{\rho}\right]
\end{align}

## Neutron Interactions

Photons tend to interact with electrons in a target atom. **Neutrons tend to interact with the nucleus.**

Neutron cross sections:

- Vary rapidly with the incident neutron energy,
- Vary erratically from one element to another
- Even vary dramatically between isotopes of the same element.

There are lots of sources of neutron cross sections. The best place to start is the Brookhaven National Laboratory National Nuclear Data Center [https://www.nndc.bnl.gov/](https://www.nndc.bnl.gov/).

Your book has a clever table (7.1) listing some of the data needed for high and low energy interaction calculations. These include:

- Elastic scattering cross sections
- Angular distribution of elastically scattered neutrons
- Inelastic scattering cross sections
- Angular distribution of inelastically scattered neutrons
- Gamma-photon yields from inelastic neutron scattering
- Resonance absorption cross sections
- Thermal-averaged absorption cross sections
- Yield of neutron-capture gamma photons
- Fission cross sections and associated gamma-photon and neutron yields

# Total cross sections

**For light nuclei** ($A<25$) and $E<1keV$, the cross section typically varies as:

\begin{align}
\sigma_t = \sigma_1 + \frac{\sigma_2}{\sqrt{E}}
\end{align}

**For solids** at energies less than about 0.01 eV, Bragg cutoffs apply. These are energies below which no coherent scattering is possible from the material's crystalline planes.

**For heavy nuclei**, the total cross section has a $\frac{1}{\sqrt{E}}$ behavior with low energy, narrow resonances and high energy broad resonances:

\begin{align}
\sigma_t \propto \frac{1}{\sqrt{E}}
\end{align}

```python
# The below IFrame displays Page 200 of your textbook:
# Shultis, J. K. (2016). Fundamentals of Nuclear Science and Engineering Third Edition,
# 3rd Edition. [Vitalsource]. Retrieved from https://bookshelf.vitalsource.com/#/books/9781498769303/
# Please take note of Figure 7.2

from IPython.display import IFrame
IFrame("https://bookshelf.vitalsource.com/books/9781498769303/pageid/222", width=1000, height=1000)

```

### Recall fission cross sections :

```python

```


```javascript
%%javascript
MathJax.Hub.Config({TeX: { equationNumbers: { autoNumber: "AMS" } }});
```

    <IPython.core.display.Javascript object>

# Maximum Entropy Principle for inference methods

The inference method based on the maximum entropy principle (**MaxEnt principle**)
asserts that the most suitable probability distribution compatible with a given set of constraints is the one with the largest entropy [Jaynes1957a, Jaynes1957b].
This method is considered as a powerful estimation technique in a wide range of probabilistic models since it brings a solution to the **universal problem** of trying to stract information from partial or incomplete data --which is usually what we have to work with--. That is the reason why it finds applications in various fields of research, beyond statistical physics, as biology [De Martino2018] and ecology [Tang2021], being also usefull for analyzing and understanding complex social or economic systems [Golan1997,Scharfenaker2020], and make financial predictions [Benedetto2015]. Problems arising from these systems are characterized by having many degrees of freedom and non-trivial interaction patterns between individual subsystems. Hence it must be dealt with
inductive inference problems due to the insufficient amount of experimental data and the incomplete nature of the information that can be extracted from them.

Moreover, the MaxEnt principle has been proven to be useful for a reasonable estimation of quantum states from incomplete data [Buzek2000,Goncalves2013,Gupta2021], where the amount of experimental resourses and time consuming make quantum tomography impractical even for an intermediate number of qubits, and therefore, approaches to validate quantum processing on these quantum devices are needed.

[De Martino2018] De Martino A, De Martino D. An introduction to the maximum entropy approach and its application to inference problems in biology. Heliyon. 2018 Apr 13;4(4):e00596. https://doi.org/10.1016/j.heliyon.2018.e00596

[Tang2021] Maximum Entropy Modeling to Predict the Impact of Climate Change on Pine Wilt Disease in China, Xinggang Tang, Yingdan Yuan, Xiangming Li and Jinchi Zhang, Front. Plant Sci., 23 April 2021. https://doi.org/10.3389/fpls.2021.652500.

[Golan1997] A. Golan, G. Judge, and D. Miller, *Maximum Entropy
Econometrics: Robust Estimation with Limited Data* (John Wiley and Sons, Chichester, United Kingdom,
1997).

[Scharfenaker2020] Scharfenaker, E., Yang, J. Maximum entropy economics. Eur. Phys. J. Spec. Top. 229, 1577–1590 (2020). https://doi.org/10.1140/epjst/e2020-000029-4

[Benedetto2015] A maximum entropy method to assess the predictability of financial and commodity prices, F.Benedetto, G.Giunta, L.Mastroeni, Digital Signal Processing
Volume 46, November 2015, Pages 19-31. https://doi.org/10.1016/j.dsp.2015.08.001

[Jaynes1957a]  E. T. Jaynes, Information theory and statistical mechanics, Physical Review **106**, 620 (1957).

[Jaynes1957b]  E. T. Jaynes, Information theory and statistical mechanics. II, Physical Review **108**, 171 (1957).

[Buzek2000]  V. Buzek and G. Drobny, Quantum tomography via the maxent principle, Journal  of Modern  Optics47,  2823 (2000). https://doi.org/10.1080/09500340008232199

[Goncalves2013]  D. Goncalves, C. Lavor, M. Gomes-Ruggiero, A. Cesario,R.  Vianna,  and  T.  Maciel,  Quantum  state  tomographywith incomplete data: Maximum entropy and variationalquantum tomography, Phys. Rev. A87, 052140 (2013). https://doi.org/10.1103/PhysRevA.87.052140

[Gupta2021] Maximal Entropy Approach for Quantum State Tomography, Rishabh Gupta, Rongxin Xia, Raphael D. Levine, and Sabre Kais, PRX QUANTUM **2**, 010318 (2021). https://doi.org/10.1103/PRXQuantum.2.010318

# Mathematical problem

Let $X$ be a random variable in a sample space $\Omega = \{x_1, \ldots, x_k\}$ with **unknown** probabilities \\(p_i=P(X=x_i), x_i ∈ \Omega\\) and $\sum_{i=1}^k p_i=1$. Mathematically, the MaxEnt formalism with \\(m\\) constraints on the expectations values $E[g_j]=\alpha_j$ of functions $g_j(x_i)$, can be expressed as a constrained optimization problem

\begin{equation}
\mathrm{max}\;S(X)\;\;
\mathrm{s.t.}
\sum_{i=1}^k p_i g_j(x_i)=\alpha_j,  ~j=1,\dots,m \;\;\;\;\;\;\;\;\;\;\;\;\;\;(1)\nonumber
\end{equation}

were $S(X)$ is the entropy of the random variable $X$. Within the information theory, $S$ is usually taken as the Shannon entroypy and then the principle gives the less biased distribution, consistent with the available data. In such a case $S(X)= H \equiv −k
\sum_{i=1}^k p_i \mathrm{log}(p_i)$. The resulting maximum entropy probability is given by:

\begin{equation}
p_i =
\frac{1}
{Z(λ_1, . . . , λ_m)}
exp \left[−λ_1 g_1(x_i) − · · · − λ_m g_m(x_i)\right],\;\;\;\;\;\;\;\;\;(2)\nonumber
\end{equation}

with $Z(λ_1, . . . , λ_m) = \sum_{i=1}^k exp [−λ_1g_1(x_i) − · · · − λ_mg_m(x_i)]$ and $λ_m$ is the Lagrangian multiplier for the $m$-th constraint given by the relation
\begin{equation}\label{classical lagrange multipliers}
\alpha_{j}= \frac{\partial}{\partial\lambda_{j}}\ln Z,\quad    1\leq j \leq n.\;\;\;\;\;\;(3)\nonumber
\end{equation}.

Hence, to find the MaxEnt probability distribution is considered a hard task due to the nonlinearities in the reconstruction algorithm. In fact, the relations in Ec. (3) represents a system of nonlinear differential equations to be solve.

# MaxEnt inference in Biology

## Inference of gene interaction networks

**Fig. 1** Inference of gene interaction networks from empirical expression data. Figure extracted from "Using the principle of entropy maximization to infer genetic interaction networks from gene expression patterns", T.R. Lezon, J.R. Banavar, M. Cieplak, A. Maritan, N.V. Fedoroff, Proc. Natl. Acad. Sci. 103 (50) (2006) 19033–19038.

# MaxEnt inference in Ecology

## Predicting the distribution of pine species and the impact of climate change on forest diseases

**Fig. 2** Habitat suitability maps showing the ocurrence of *P. desinflora* by 2050 and 2070 under two distinct climate change scenarios in China.  Figure extracted from "Maximum Entropy Modeling to Predict the Impact of Climate Change on Pine Wilt Disease in China", Xinggang Tang, Yingdan Yuan, Xiangming Li and Jinchi Zhang, Front. Plant Sci., 23 April 2021.

# Solving MaxEnt as a QUBO problem

The Quadratic Unconstrained Binary Optimization (QUBO) [Kochenberger2014] is a model for representing a wide range of combinatorial optimization problems. Moreover, due to its close connection to Ising models, QUBO constitutes a central problem class for Adiabatic quantum computation feasible to be solved through quantum annealing.

If $f_Q(x)=x^TQx$ is a quadratic polinomial over binary variables $x_i\in B=\{0,1\}$, where $Q\in \mathbb {R} ^{n\times n}$ is a symmetric $n\times n$ matrix, the QUBO problem consists of finding a binary vector $x^{*}$ that minimize $f_Q$.

## Goal

We have redefined the MaxEnt problem as a QUBO problem $\left(P^TQP+C^TP\right)$. For this purpose, the first step is to find an appropriate entropy function $S$ and codified the variables to be obtained as a result of the minimization process, in our case the probabilities $p_i$, as a binary vector. Expanding the Shannon's entropy $H$ to first order in the distribution $P=(p_1,p_1,\dots,p_k)^T$, we obtain the quadratic entropy

\begin{equation}
H(P) \approx −k
\sum_{i=1}^k p_i \mathrm{log}(p_i)=2 - 2 P^TP,\;\;\;\;\;\;\;\;\;\; (4)\nonumber
\label{entropy}
\end{equation}

were we have take the value $k=2$ according to a random variable $X\in\Omega=\{0,1\}$.
Then, we are interesting in to find the probability distribution $P$ that satisfies the constraints in Eq. $\left(1\right)$ and maximizes the quadratic entropy $\left(4\right)$.

$1.$  We define the cost function $f(P)$ as:
\begin{equation}
f(P) = -H(P) + \sum_{j= 0}^{m} (G_j^T P - \alpha_j) ^2.\;\;\;\;\;\;\;\;\;\; (5)\nonumber
\end{equation}

where $G_j=(g_j(x_1),g_j(x_2),\dots,g_j(x_k))^T$ is the vector which contains the image of the function $g_j\in \{g_1,\dots,g_m\}$. After trivial algebra manipulation Eq. $(5)$ is reduced to

\begin{align}
f(P)=-2 + \sum_{j= 0}^{m}  \alpha_j^2 +  \left(\sum_{j= 0}^{m}(-2) \alpha_j G_j^T\right) P + P^T \left( 2I_k  +\sum_{j= 0}^{m} G_j G_j^T\right) P \;\;\;\;\;\;\;\;\;\; (6)\nonumber
 \end{align}

Then, $f(P)$ can be rewritten as
\begin{equation}
f(P) =  P^T Q P + C^T P + cte, \;\;\;\;\;\;\;\;\;\; (7)\nonumber
\end{equation}

with $C^T = \sum_{j= 0}^{m}(-2) \alpha_j G_j^T$, and $Q = 2 I_n  +\sum_{j= 0}^{m} G_j G_j^T$.

$2.$ Now we will express each entry $p_i$ of the probability distribution in a binary basis, i.e.,
\begin{equation}
p_i  = \sum_{k=1}^d \frac{a_{ik}}{2^k},\;\;\;\;\;\;\;\;\;\; (8)\nonumber
\end{equation}
where each $a_{ik}$ can be $0$ or $1$. For a given $d$ we have a restriction of the values that we able to represent $(0 \leq p_i \leq 1- 1/2^d$ with precision $1/2^d)$. Then,

\begin{align}
p_i  &= (\frac{1}{2}, \ldots,\frac{1}{2^d})  (a_{i1}, \ldots, a_{id})^T\;\;\;\;\;\;\;\;\;\; (9)\nonumber\\
\mathrm{or}\;\; P & = S a,\nonumber
\end{align}

with $S$ an adequate matrix that performs the transformation.

Finally, we obtain the cost function in a suitable form to be solve as a QUBO problem

\begin{equation}
f(p) =  cte + C^T S a + a^T S^TQS a.
\end{equation}

[Kochenberger2014] Kochenberger, Gary; Hao, Jin-Kao (2014). "The unconstrained binary quadratic programming problem: a survey" (PDF). Journal of Combinatorial Optimization. **28**: 58–81. http://doi:10.1007/s10878-014-9734-0.

# A puzzlelike problem

The following code finds the MaxEnt probability distribution given the appearance frequencies of the faces of a die as constraints.

```python
import numpy as np
import function as f
```

```python
#Example 1: Dice without constraints
nb = 6             #Number of bits
lam = 0.001        #Optimization constant
numreads = 10000   #number of reads
y0= np.array ([[1.0],[1.0],[1.0],[1.0],[1.0], [1.0] ])
alpha =  np.array ([1.0 ])
y = [y0]
x,p,cost, cost_bin = f.solution(y, alpha, nb, lam,numreads )
print('Binary solution: ', x)
print('Probability: ', p, 'Sum: ', sum(p))
print('Cost: ', cost[0])
```

    Binary solution:  [0 0 1 0 1 0 0 0 1 1 0 0 0 0 1 1 0 0 0 0 1 0 1 1 0 0 1 0 0 1 0 0 1 0 1 0]
    Probability:  [0.15625  0.1875   0.1875   0.171875 0.140625 0.15625 ] Sum:  1.0
    Cost:  -0.9996630859375

```python
#Example 2: Dice with fair mean value
nb = 6             #Number of bits
lam = 0.001        #Optimization constant
numreads = 10000   #number of reads
y0= np.array ([[1.0],[1.0],[1.0],[1.0],[1.0], [1.0] ])
y1= np.array ([[1.0],[2.0],[3.0],[4.0],[5.0], [6.0] ])
alpha =  np.array ([1.0, 3.5 ])
y = [y0, y1]
x,p,cost, cost_bin = f.solution(y, alpha, nb, lam,numreads )
print('Binary solution: ', x)
print('Probability: ', p, 'Sum: ', sum(p))
print('Cost: ', cost[0])
```

    Binary solution:  [0 0 1 0 1 0 0 0 1 0 1 1 0 0 1 0 1 1 0 0 1 0 1 1 0 0 1 0 1 1 0 0 1 0 1 0]
    Probability:  [0.15625  0.171875 0.171875 0.171875 0.171875 0.15625 ] Sum:  1.0
    Cost:  -13.249666015625

```python
#Example 3: Loaded dice
nb = 8             #Number of bits
lam = 0.0001       #Optimization constant
numreads = 50000   #number of reads
y0= np.array ([[1.0],[1.0],[1.0],[1.0],[1.0], [1.0] ])
y1= np.array ([[1.0],[2.0],[3.0],[4.0],[5.0], [6.0] ])
alpha =  np.array ([1.0, 6 ])
y = [y0, y1]
x,p,cost, cost_bin = f.solution(y, alpha, nb, lam,numreads )
print('Binary solution: ', x)
print('Probability: ', p, 'Sum: ', sum(p))
print('Cost: ', cost[0])
```

    Binary solution:  [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0
     0 1 0 1 1 1 1 1 0 1 0]
    Probability:  [0.        0.        0.        0.0234375 0.0078125 0.9765625] Sum:  1.0078125
    Cost:  -36.99968707275391

```python
#Example 4: Dice with one face with fixed probability
nb = 8             #Number of bits
lam = 0.01        #Optimization constant
numreads = 10000   #number of reads
y0= np.array ([[1.0],[1.0],[1.0],[1.0],[1.0], [1.0] ])
y1= np.array ([[0.0],[1.0],[0.0],[0.0],[0.0], [0.0] ])
alpha =  np.array ([1.0, 0.8 ])
y = [y0, y1]
x,p,cost, cost_bin = f.solution(y, alpha, nb, lam,numreads )
print('Binary solution: ', x)
print('Probability: ', p, 'Sum: ', sum(p))
print('Cost: ', cost[0])
```

    Binary solution:  [0 0 0 0 1 1 0 0 1 1 0 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 1 0 1 1 0 0 0 0 1
     0 1 0 0 0 0 0 1 1 0 0]
    Probability:  [0.046875   0.78515625 0.0390625  0.04296875 0.0390625  0.046875  ] Sum:  1.0
    Cost:  -1.6272644042968751

```python
#Example 5: Dice with two faces summing a fixed probability
nb = 8             #Number of bits
lam = 0.01        #Optimization constant
numreads = 10000   #number of reads
y0= np.array ([[1.0],[1.0],[1.0],[1.0],[1.0], [1.0] ])
y1= np.array ([[1.0],[1.0],[0.0],[0.0],[0.0], [0.0] ])
alpha =  np.array ([1.0, 0.7 ])
y = [y0, y1]
x,p,cost, cost_bin = f.solution(y, alpha, nb, lam,numreads )
print('Binary solution: ', x)
print('Probability: ', p, 'Sum: ', sum(p))
print('Cost: ', cost[0])
```

    Binary solution:  [0 1 0 1 1 0 0 1 0 1 0 1 1 0 0 1 0 0 0 1 0 0 1 1 0 0 0 1 0 0 1 1 0 0 0 1 0
     1 0 0 0 0 0 1 0 1 0 0]
    Probability:  [0.34765625 0.34765625 0.07421875 0.07421875 0.078125   0.078125  ] Sum:  1.0
    Cost:  -1.4846789550781248


Natural Language Processing For Everyone

Text Representation

Bruno Gonçalves

        www.data4sci.com

        @bgoncalves, @data4sci

In this lesson we will see in some details how we can best represent text in our application. Let's start by importing the modules we will be using:

```python
import string
from collections import Counter
from pprint import pprint
import gzip

import matplotlib
import matplotlib.pyplot as plt
import numpy as np

import watermark

%matplotlib inline
%load_ext watermark
```

List out the versions of all loaded libraries

```python
%watermark -n -v -m -g -iv
```

    Python implementation: CPython
    Python version       : 3.8.5
    IPython version      : 7.19.0

    Compiler    : Clang 10.0.0
    OS          : Darwin
    Release     : 20.3.0
    Machine     : x86_64
    Processor   : i386
    CPU cores   : 16
    Architecture: 64bit

    Git hash: 842cbaa9fb86ca89575a80bfaea9a8abcdb598ac

    matplotlib: 3.3.2
    numpy     : 1.20.1
    json      : 2.0.9
    watermark : 2.1.0

Set the default style

```python
plt.style.use('./d4sci.mplstyle')
```

We choose a well known nursery rhyme, that has the added distinction of having been the first audio ever recorded, to be the short snippet of text that we will use in our examples:

```python
text = """Mary had a little lamb, little lamb,
    little lamb. 'Mary' had a little lamb
    whose fleece was white as snow.
    And everywhere that Mary went
    Mary went, MARY went. Everywhere
    that mary went,
    The lamb was sure to go"""
```

## Tokenization

The first step in any analysis is to tokenize the text. What this means is that we will extract all the individual words in the text. For the sake of simplicity, we will assume that our text is well formed and that our words are delimited either by white space or punctuation characters.

```python
print(string.punctuation)
```

    !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

```python
def extract_words(text):
    temp = text.split() # Split the text on whitespace
    text_words = []

    for word in temp:
        # Remove any punctuation characters present in the beginning of the word
        while word[0] in string.punctuation:
            word = word[1:]

        # Remove any punctuation characters present in the end of the word
        while word[-1] in string.punctuation:
            word = word[:-1]

        # Append this word into our list of words.
        text_words.append(word.lower())

    return text_words
```

After this step we now have our text represented as an array of individual, lowercase, words:

```python
text_words = extract_words(text)
print(text_words)
```

    ['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb', 'mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went', 'everywhere', 'that', 'mary', 'went', 'the', 'lamb', 'was', 'sure', 'to', 'go']

As we saw during the video, this is a wasteful way to represent text. We can be much more efficient by representing each word by a number

```python
word_dict = {}
word_list = []
vocabulary_size = 0
text_tokens = []

for word in text_words:
    # If we are seeing this word for the first time, create an id for it and added it to our word dictionary
    if word not in word_dict:
        word_dict[word] = vocabulary_size
        word_list.append(word)
        vocabulary_size += 1

    # add the token corresponding to the current word to the tokenized text.
    text_tokens.append(word_dict[word])
```

When we were tokenizing our text, we also generated a dictionary **word_dict** that maps words to integers and a **word_list** that maps each integer to the corresponding word.

```python
print("Word list:", word_list, "\n\n Word dictionary:")
pprint(word_dict)
```

    Word list: ['mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'went', 'the', 'sure', 'to', 'go']

     Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

These two datastructures already proved their usefulness when we converted our text to a list of tokens.

```python
print(text_tokens)
```

    [0, 1, 2, 3, 4, 3, 4, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 14, 0, 14, 0, 14, 12, 13, 0, 14, 15, 4, 7, 16, 17, 18]

Unfortunately, while this representation is convenient for memory reasons it has some severe limitations. Perhaps the most important of which is the fact that computers naturally assume that numbers can be operated on mathematically (by addition, subtraction, etc) in a way that doesn't match our understanding of words.

## One-hot encoding

One typical way of overcoming this difficulty is to represent each word by a one-hot encoded vector where every element is zero except the one corresponding to a specific word.

```python
def one_hot(word, word_dict):
    """
        Generate a one-hot encoded vector corresponding to *word*
    """

    vector = np.zeros(len(word_dict))
    vector[word_dict[word]] = 1

    return vector
```

So, for example, the word "fleece" would be represented by:

```python
print(vocabulary_size)
print(len(word_dict))
```

    19
    19

```python
fleece_hot = one_hot("fleece", word_dict)
print(fleece_hot)
```

    [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

This vector has every element set to zero, except element 6, since:

```python
print(word_dict["fleece"])
fleece_hot[6] == 1
```

    6

    True

```python
print(fleece_hot.sum())
```

    1.0

## Bag of words

We can now use the one-hot encoded vector for each word to produce a vector representation of our original text, by simply adding up all the one-hot encoded vectors:

```python
text_vector1 = np.zeros(vocabulary_size)

for word in text_words:
    hot_word = one_hot(word, word_dict)
    text_vector1 += hot_word

print(text_vector1)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

In practice, we can also easily skip the encoding step at the word level by using the *word_dict* defined above:

```python
text_vector = np.zeros(vocabulary_size)

for word in text_words:
    text_vector[word_dict[word]] += 1

print(text_vector)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

Naturally, this approach is completely equivalent to the previous one and has the added advantage of being more efficient in terms of both speed and memory requirements.

This is known as the __bag of words__ representation of the text. It should be noted that these vectors simply contains the number of times each word appears in our document, so we can easily tell that the word *mary* appears exactly 6 times in our little nursery rhyme.

```python
text_vector[word_dict["mary"]]
```

    6.0

A more pythonic (and efficient) way of producing the same result is to use the standard __Counter__ module:

```python
word_counts = Counter(text_words)
pprint(word_counts)
```

    Counter({'mary': 6,
             'lamb': 5,
             'little': 4,
             'went': 4,
             'had': 2,
             'a': 2,
             'was': 2,
             'everywhere': 2,
             'that': 2,
             'whose': 1,
             'fleece': 1,
             'white': 1,
             'as': 1,
             'snow': 1,
             'and': 1,
             'the': 1,
             'sure': 1,
             'to': 1,
             'go': 1})

From which we can easily generate the __text_vector__ and __word_dict__ data structures:

```python
items = list(word_counts.items())

# Extract word dictionary and vector representation
word_dict2 = dict([[items[i][0], i] for i in range(len(items))])
text_vector2 = [items[i][1] for i in range(len(items))]
```

```python
word_counts['mary']
```

    6

And let's take a look at them:

```python
text_vector
```

    array([6., 2., 2., 4., 5., 1., 1., 2., 1., 1., 1., 1., 2., 2., 4., 1., 1.,
           1., 1.])

```python
print("Text vector:", text_vector2, "\n\nWord dictionary:")
pprint(word_dict2)
```

    Text vector: [6, 2, 2, 4, 5, 1, 1, 2, 1, 1, 1, 1, 2, 2, 4, 1, 1, 1, 1]

    Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

The results using this approach are slightly different than the previous ones, because the words are mapped to different integer ids but the corresponding values are the same:

```python
for word in word_dict.keys():
    if text_vector[word_dict[word]] != text_vector2[word_dict2[word]]:
        print("Error!")
```

As expected, there are no differences!

## Term Frequency

The bag of words vector representation introduced above relies simply on the frequency of occurence of each word. Following a long tradition of giving fancy names to simple ideas, this is known as __Term Frequency__.

Intuitively, we expect the the frequency with which a given word is mentioned should correspond to the relevance of that word for the piece of text we are considering. For example, **Mary** is a pretty important word in our little nursery rhyme and indeed it is the one that occurs the most often:

```python
sorted(items, key=lambda x:x[1], reverse=True)
```

    [('mary', 6),
     ('lamb', 5),
     ('little', 4),
     ('went', 4),
     ('had', 2),
     ('a', 2),
     ('was', 2),
     ('everywhere', 2),
     ('that', 2),
     ('whose', 1),
     ('fleece', 1),
     ('white', 1),
     ('as', 1),
     ('snow', 1),
     ('and', 1),
     ('the', 1),
     ('sure', 1),
     ('to', 1),
     ('go', 1)]

However, it's hard to draw conclusions from such a small piece of text. Let us consider a significantly larger piece of text, the first 100 MB of the english Wikipedia from: http://mattmahoney.net/dc/textdata. For the sake of convenience, text8.gz has been included in this repository in the **data/** directory. We start by loading it's contents into memory as an array of words:

```python
data = []

for line in gzip.open("data/text8.gz", 'rt'):
    data.extend(line.strip().split())
```

Now let's take a look at the first 50 words in this large corpus:

```python
data[:50]
```

    ['anarchism',
     'originated',
     'as',
     'a',
     'term',
     'of',
     'abuse',
     'first',
     'used',
     'against',
     'early',
     'working',
     'class',
     'radicals',
     'including',
     'the',
     'diggers',
     'of',
     'the',
     'english',
     'revolution',
     'and',
     'the',
     'sans',
     'culottes',
     'of',
     'the',
     'french',
     'revolution',
     'whilst',
     'the',
     'term',
     'is',
     'still',
     'used',
     'in',
     'a',
     'pejorative',
     'way',
     'to',
     'describe',
     'any',
     'act',
     'that',
     'used',
     'violent',
     'means',
     'to',
     'destroy',
     'the']

And the top 10 most common words

```python
counts = Counter(data)

sorted_counts = sorted(list(counts.items()), key=lambda x: x[1], reverse=True)

for word, count in sorted_counts[:10]:
    print(word, count)
```

    the 1061396
    of 593677
    and 416629
    one 411764
    in 372201
    a 325873
    to 316376
    zero 264975
    nine 250430
    two 192644

Surprisingly, we find that the most common words are not particularly meaningful. Indeed, this is a common occurence in Natural Language Processing. The most frequent words are typically auxiliaries required due to gramatical rules.

On the other hand, there is also a large number of words that occur very infrequently as can be easily seen by glancing at the word freqency distribution.

```python
dist = Counter(counts.values())
dist = list(dist.items())
dist.sort(key=lambda x:x[0])
dist = np.array(dist)

norm = np.dot(dist.T[0], dist.T[1])

plt.loglog(dist.T[0], dist.T[1]/norm)
plt.xlabel("count")
plt.ylabel("P(count)")
plt.title("Word frequency distribution")
plt.gcf().set_size_inches(11, 8)
```

## Stopwords

One common technique to simplify NLP tasks is to remove what are known as Stopwords, words that are very frequent but not meaningful. If we simply remove the most common 100 words, we significantly reduce the amount of data we have to consider while losing little information.

```python
stopwords = set([word for word, count in sorted_counts[:100]])

clean_data = []

for word in data:
    if word not in stopwords:
        clean_data.append(word)

print("Original size:", len(data))
print("Clean size:", len(clean_data))
print("Reduction:", 1-len(clean_data)/len(data))
```

    Original size: 17005207
    Clean size: 9006229
    Reduction: 0.470384041782026

```python
clean_data[:50]
```

    ['anarchism',
     'originated',
     'term',
     'abuse',
     'against',
     'early',
     'working',
     'class',
     'radicals',
     'including',
     'diggers',
     'english',
     'revolution',
     'sans',
     'culottes',
     'french',
     'revolution',
     'whilst',
     'term',
     'still',
     'pejorative',
     'way',
     'describe',
     'any',
     'act',
     'violent',
     'means',
     'destroy',
     'organization',
     'society',
     'taken',
     'positive',
     'label',
     'self',
     'defined',
     'anarchists',
     'word',
     'anarchism',
     'derived',
     'greek',
     'without',
     'archons',
     'ruler',
     'chief',
     'king',
     'anarchism',
     'political',
     'philosophy',
     'belief',
     'rulers']

Wow, our dataset size was reduced almost in half!

In practice, we don't simply remove the most common words in our corpus but rather a manually curate list of stopwords. Lists for dozens of languages and applications can easily be found online.

## Term Frequency/Inverse Document Frequency

One way of determining of the relative importance of a word is to see how often it appears across multiple documents. Words that are relevant to a specific topic are more likely to appear in documents about that topic and much less in documents about other topics. On the other hand, less meaningful words (like **the**) will be common across documents about any subject.

To measure the document frequency of a word we will need to have multiple documents. For the sake of simplicity, we will treat each sentence of our nursery rhyme as an individual document:

```python
print(text)
```

    Mary had a little lamb, little lamb,
        little lamb. 'Mary' had a little lamb
        whose fleece was white as snow.
        And everywhere that Mary went
        Mary went, MARY went. Everywhere
        that mary went,
        The lamb was sure to go

```python
corpus_text = text.split('.')
corpus_words = []

for document in corpus_text:
    doc_words = extract_words(document)
    corpus_words.append(doc_words)
```

Now our corpus is represented as a list of word lists, where each list is just the word representation of the corresponding sentence:

```python
print(len(corpus_words))
```

    4

```python
pprint(corpus_words)
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

Let us now calculate the number of documents in which each word appears:

```python
document_count = {}

for document in corpus_words:
    word_set = set(document)

    for word in word_set:
        document_count[word] = document_count.get(word, 0) + 1

pprint(document_count)
```

    {'a': 2,
     'and': 1,
     'as': 1,
     'everywhere': 2,
     'fleece': 1,
     'go': 1,
     'had': 2,
     'lamb': 3,
     'little': 2,
     'mary': 4,
     'snow': 1,
     'sure': 1,
     'that': 2,
     'the': 1,
     'to': 1,
     'was': 2,
     'went': 2,
     'white': 1,
     'whose': 1}

As we can see, the word __Mary__ appears in all 4 of our documents, making it useless when it comes to distinguish between the different sentences. On the other hand, words like __white__ which appear in only one document are very discriminative. Using this approach we can define a new quantity, the ___Inverse Document Frequency__ that tells us how frequent a word is across the documents in a specific corpus:

```python
def inv_doc_freq(corpus_words):
    number_docs = len(corpus_words)

    document_count = {}

    for document in corpus_words:
        word_set = set(document)

        for word in word_set:
            document_count[word] = document_count.get(word, 0) + 1

    IDF = {}

    for word in document_count:
        IDF[word] = np.log(number_docs/document_count[word])

    return IDF
```

Where we followed the convention of using the logarithm of the inverse document frequency. This has the numerical advantage of avoiding to have to handle small fractional numbers.

We can easily see that the IDF gives a smaller weight to the most common words and a higher weight to the less frequent:

```python
corpus_words
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

```python
IDF = inv_doc_freq(corpus_words)

pprint(IDF)
```

    {'a': 0.6931471805599453,
     'and': 1.3862943611198906,
     'as': 1.3862943611198906,
     'everywhere': 0.6931471805599453,
     'fleece': 1.3862943611198906,
     'go': 1.3862943611198906,
     'had': 0.6931471805599453,
     'lamb': 0.28768207245178085,
     'little': 0.6931471805599453,
     'mary': 0.0,
     'snow': 1.3862943611198906,
     'sure': 1.3862943611198906,
     'that': 0.6931471805599453,
     'the': 1.3862943611198906,
     'to': 1.3862943611198906,
     'was': 0.6931471805599453,
     'went': 0.6931471805599453,
     'white': 1.3862943611198906,
     'whose': 1.3862943611198906}

As expected **Mary** has the smallest weight of all words 0, meaning that it is effectively removed from the dataset. You can consider this as a way of implicitly identify and remove stopwords. In case you do want to keep even the words that appear in every document, you can just add a 1. to the argument of the logarithm above:

\begin{equation}
\log\left[1+\frac{N_d}{N_d\left(w\right)}\right]
\end{equation}

When we multiply the term frequency of each word by it's inverse document frequency, we have a good way of quantifying how relevant a word is to understand the meaning of a specific document.

```python
def tf_idf(corpus_words):
    IDF = inv_doc_freq(corpus_words)

    TFIDF = []

    for document in corpus_words:
        TFIDF.append(Counter(document))

    for document in TFIDF:
        for word in document:
            document[word] = document[word]*IDF[word]

    return TFIDF
```

```python
tf_idf(corpus_words)
```

    [Counter({'mary': 0.0,
              'had': 0.6931471805599453,
              'a': 0.6931471805599453,
              'little': 2.0794415416798357,
              'lamb': 0.8630462173553426}),
     Counter({'mary': 0.0,
              'had': 0.6931471805599453,
              'a': 0.6931471805599453,
              'little': 0.6931471805599453,
              'lamb': 0.28768207245178085,
              'whose': 1.3862943611198906,
              'fleece': 1.3862943611198906,
              'was': 0.6931471805599453,
              'white': 1.3862943611198906,
              'as': 1.3862943611198906,
              'snow': 1.3862943611198906}),
     Counter({'and': 1.3862943611198906,
              'everywhere': 0.6931471805599453,
              'that': 0.6931471805599453,
              'mary': 0.0,
              'went': 2.0794415416798357}),
     Counter({'everywhere': 0.6931471805599453,
              'that': 0.6931471805599453,
              'mary': 0.0,
              'went': 0.6931471805599453,
              'the': 1.3862943611198906,
              'lamb': 0.28768207245178085,
              'was': 0.6931471805599453,
              'sure': 1.3862943611198906,
              'to': 1.3862943611198906,
              'go': 1.3862943611198906})]

Now we finally have a vector representation of each of our documents that takes the informational contributions of each word into account. Each of these vectors provides us with a unique representation of each document, in the context (corpus) in which it occurs, making it posssible to define the similarity of two documents, etc.

## Porter Stemmer

There is still, however, one issue with our approach to representing text. Since we treat each word as a unique token and completely independently from all others, for large documents we will end up with many variations of the same word such as verb conjugations, the corresponding adverbs and nouns, etc.

One way around this difficulty is to use stemming algorithm to reduce words to their root (or stem) version. The most famous Stemming algorithm is known as the **Porter Stemmer** and was introduced by Martin Porter in 1980 [Program 14, 130 (1980)](https://dl.acm.org/citation.cfm?id=275705)

The algorithm starts by defining consonants (C) and vowels (V):

```python
V = set('aeiouy')
C = set('bcdfghjklmnpqrstvwxz')
```

The stem of a word is what is left of that word after a speficic ending has been removed. A function to do this is easy to implement:

```python
def get_stem(suffix, word):
    """
        Extract the stem of a word
    """

    if word.lower().endswith(suffix.lower()): # Case insensitive comparison
        return word[:-len(suffix)]

    return None
```

It also defines words (or stems) to be sequences of vowels and consonants of the form:

\begin{equation}
[C](VC)^m[V]
\end{equation}

where $m$ is called the **measure** of the word and [] represent optional sections.

```python
def measure(orig_word):
    """
        Calculate the "measure" m of a word or stem, according to the Porter Stemmer algorthim
    """

    word = orig_word.lower()

    optV = False
    optC = False
    VC = False

    m = 0
    pos = 0

    # We can think of this implementation as a simple finite state machine
    # looks for sequences of vowels or consonants depending of the state
    # in which it's in, while keeping track of how many VC sequences it
    # has encountered.
    # The presence of the optional V and C portions is recorded in the
    # optV and optC booleans.

    # We're at the initial state.
    # gobble up all the optional consonants at the beginning of the word
    while pos < len(word) and word[pos] in C:
        pos += 1
        optC = True

    while pos < len(word):
        # Now we know that the next state must be a vowel
        while pos < len(word) and word[pos] in V:
            pos += 1
            optV = True

        # Followed by a consonant
        while pos < len(word) and word[pos] in C:
            pos += 1
            optV = False

        # If a consonant was found, then we matched VC
        # so we should increment m by one. Otherwise,
        # optV remained true and we simply had a dangling
        # V sequence.
        if not optV:
            m += 1

    return m
```

Let's consider a simple example. The word __crepusculars__ should have measure 4:

[cr] (ep) (usc) (ul) (ars)

and indeed it does.

```python
word = "crepusculars"
print(measure(word))
```

    4

(agr) = (VC)

```python
word = "agr"
print(measure(word))
```

    1

The Porter algorithm sequentially applies a series of transformation rules over a series of 5 steps (step 1 is divided in 3 substeps and step 5 in 2). The rules are only applied if a certain condition is true.

In addition to possibily specifying a requirement on the measure of a word, conditions can make use of different boolean functions as well:

```python
def ends_with(char, stem):
    """
        Checks the ending of the word
    """
    return stem[-1] == char

def double_consonant(stem):
    """
        Checks the ending of a word for a double consonant
    """
    if len(stem) < 2:
        return False

    if stem[-1] in C and stem[-2] == stem[-1]:
        return True

    return False

def contains_vowel(stem):
    """
        Checks if a word contains a vowel or not
    """
    return len(set(stem) & V) > 0
```

Finally, we define a function to apply a specific rule to a word or stem:

```python
def apply_rule(condition, suffix, replacement, word):
    """
        Apply Porter Stemmer rule.
        if "condition" is True replace "suffix" by "replacement" in "word"
    """

    stem = get_stem(suffix, word)

    if stem is not None and condition is True:
        # Remove the suffix
        word = stem

        # Add the replacement suffix, if any
        if replacement is not None:
            word += replacement

    return word
```

Now we can see how rules can be applied. For example, this rule, from step 1b is successfully applied to __pastered__:

```python
word = "plastered"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'plaster'

```python
stem
```

    'plaster'

```python
contains_vowel(stem)
```

    True

While try applying the same rule to **bled** will fail to pass the condition resulting in no change.

```python
word = "bled"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'bled'

```python
stem
```

    'bl'

```python
contains_vowel(stem)
```

    False

For a more complex example, we have, in Step 4:

```python
word = "adoption"
suffix = "ion"
stem = get_stem(suffix, word)
apply_rule(measure(stem) > 1 and (ends_with("s", stem) or ends_with("t", stem)), suffix, None, word)
```

    'adopt'

```python
ends_with("t", stem)
```

    True

```python
ends_with("s", stem)
```

    False

```python
measure(stem)
```

    2

In total, the Porter Stemmer algorithm (for the English language) applies several dozen rules (see https://tartarus.org/martin/PorterStemmer/def.txt for a complete list). Implementing all of them is both tedious and error prone, so we abstain from providing a full implementation of the algorithm here. High quality implementations can be found in all major NLP libraries such as [NLTK](http://www.nltk.org/howto/stem.html).

The dificulties of defining matching rules to arbitrary text cannot be fully resolved without the use of Regular Expressions (typically implemented as Finite State Machines like our __measure__ implementation above), a more advanced topic that is beyond the scope of this course.