# The One-Dimensional Particle in a Box

## 🥅 Learning Objectives
- Determine the energies and eigenfunctions of the particle-in-a-box.
- Learn how to normalize a wavefunction.
- Learn how to compute expectation values for quantum-mechanical operators.
- Learn the postulates of quantum mechanics

## Cyanine Dyes

Cyanine dye molecules are often modelled as one-dimension particles in a box. To understand why, start by thinking classically.  You learn in organic chemistry that electrons can more “freely” along alternating double bonds.  If this is true, then you can imagine that the electrons can more from one Nitrogen to the other, almost without resistance.  On the other hand, there are sp3-hybridized functional groups attached to the Nitrogen atom, so once the electron gets to Nitrogen atom, it has to turn around and go back whence it came.   A very, very, very simple model would be to imagine that the electron is totally free between the Nitrogen atoms, and totally forbidden from going much beyond the Nitrogen atoms.  This suggests modeling these systems a potential energy function like:

$$
V(x) =
\begin{cases}
    +\infty & x\leq 0\\
    0       & 0\lt x \lt a\\
    +\infty & a \leq x
\end{cases}
$$

where $a$ is the length of the box. A reasonable approximate formula for $a$ is

$$
a = \left(5.67 + 2.49 (k + 1)\right) \cdot 10^{-10} \text{ m}
$$

## Postulate: The squared magnitude of the wavefunction is proportional to probability
What is the interpretation of the wavefunction? The Born postulate indicates that the squared magnitude of the wavefunction is proportional to the probability of observing the system at that location. E.g., if $\psi(x)$ is the wavefunction for an electron as a function of $x$, then

$$
p(x) = |\psi(x)|^2
$$

is the probability of observing an electron at the point $x$. This is called the Born Postulate.

## The Wavefunctions of the Particle in a Box (boundary conditions)

The nice thing about this “particle in a box” model is that it is easy to solve the time-independent Schrödinger equation in this case.  Because there is no chance that the particle could ever “escape” an infinite box like this (such an electron would have infinite potential energy!), $|\psi(x)|^2$ must equal zero outside the box. Therefore the wavefunction can only be nonzero inside the box. In addition, the wavefunction should be zero at the edges of the box, because otherwise the wavefunction will not be continuous.  So we should have a wavefunction like

$$
\psi(x) =
\begin{cases}
    0& x\le 0\\
    \text{?????}    & 0 < x < a\\
    0 & a \le x
\end{cases}
$$

## Postulate: The wavefunction of a system is determined by solving the Schrödinger equation
How do we find the wavefunction for the particle-in-a-box or, for that matter, any other system? The wavefunction can be determined by solving the time-independent (when the potential is time-independent) or time-dependent (when the potential is time-dependent) Schrödinger equation.

## The Wavefunctions of the Particle in a Box (solution)

To find the wavefunctions for a system, one solves the Schrödinger equation. For a particle of mass $m$ in a one-dimensional box, the (time-independent) Schrödinger equation is:

$$
\left(-\frac{\hbar^2}{2m} \frac{d^2}{dx^2} + V(x) \right)\psi_n(x) = E_n \psi_n(x)
$$

where

$$
V(x) =
\begin{cases}
    +\infty & x\leq 0\\
    0       & 0\lt x \lt a\\
    +\infty & a \leq x
\end{cases}
$$

We already deduced that $\psi(x) = 0$ except when the electron is inside the box ($0 < x  Question: What function(s), when differentiated twice, are proportional to themselves?

This suggests that the eigenfunctions of the 1-dimensional particle-in-a-box must be some linear combination of sine and cosine functions,

$$
\psi_n(x) = A \sin(cx) + B  \cos(dx)
$$

We know that the wavefunction must be zero at the edges of the box, $\psi(0) = 0$ and $\psi(a) = 0$. These are called the *boundary conditions* for the problem. Examining the first boundary condition,

$$
0 = \psi(0) = A \sin(0) + B \cos(0) = 0 + B
$$

indicates that $B=0$. The second boundary condition

$$
0 = \psi(a) = A \sin(ca)
$$

requires us to recall that $\sin(x) = 0$ whenever $x$ is an integer multiple of $\pi$. So $c=n\pi$ where $n=1,2,3,\ldots$. The wavefunction for the particle in a box is thus,

$$
\psi_n(x) = A_n \sin\left(\tfrac{n \pi x}{a}\right) \qquad \qquad n=1,2,3,\ldots
$$

## Normalization of Wavefunctions
As seen in the previous section, if a wavefunction solves the Schrödinger equation, any constant multiple of the wavefunction also solves the Schrödinger equation,

$$
\hat{H} \psi(x) = E \psi(x) \quad \longleftrightarrow \quad \hat{H} \left(A\psi(x)\right) = E \left(A\psi(x)\right)
$$

Owing to the Born postulate, the complex square of the wavefunction can be interpreted as probability. Since the probability of a particle being at *some* point in space is one, we can define the normalization constant, $A$, for the wavefunction through the requirement that:

$$
\int_{-\infty}^{\infty} \left|\psi(x)\right|^2 dx = 1.
$$

In the case of a particle in a box, this is:

$$
\begin{align}
1 &= \int_{-\infty}^{\infty} \left|\psi_n(x)\right|^2 dx  \\
&= \int_0^a \psi_n(x) \psi_n^*(x) dx \\
&= \int_0^a A_n \sin\left(\tfrac{n \pi x}{a}\right) \left(A_n \sin\left(\tfrac{n \pi x}{a}\right) \right)^* dx \\
&= \left|A_n\right|^2\int_0^a \sin^2\left(\tfrac{n \pi x}{a}\right) dx
\end{align}
$$

To evaluate this integral, it is useful to remember some [trigonometric identities](https://en.wikipedia.org/wiki/List_of_trigonometric_identities). (You can learn more about how I remember trigonometric identities [here](../linkedFiles/TrigIdentities.md).) The specific identity we need here is $\sin^2 x = \tfrac{1}{2}(1-\cos 2x)$:

$$
\begin{align}
1 &= \left|A_n\right|^2\int_0^a \sin^2\left(\tfrac{n \pi x}{a}\right) \,dx \\
 &= \left|A_n\right|^2\int_0^a \tfrac{1}{2}\left(1-\cos \left(\tfrac{2n \pi x}{a}\right)\right) \,dx \\
 &=\tfrac{\left|A_n\right|^2}{2} \left( \int_0^a 1 \,dx - \int_0^a \cos \left(\tfrac{2n \pi x}{a}\right)\,dx  \right) \\
 &=\tfrac{\left|A_n\right|^2}{2} \left( \left[ x \right]_0^a - \left[\frac{-a}{2 n \pi}\sin \left(\tfrac{2n \pi x}{a}\right) \right]_0^a  \right) \\
 &=\tfrac{\left|A_n\right|^2}{2} \left( a - 0  \right)
\end{align}
$$

So

$$
 \left|A_n\right|^2 = \tfrac{2}{a}
$$

Note that this does not completely determine $A_n$. For example, any of the following normalization constants are allowed,

$$
A_n = \sqrt{\tfrac{2}{a}}
= - \sqrt{\tfrac{2}{a}}
= i \sqrt{\tfrac{2}{a}}
= -i \sqrt{\tfrac{2}{a}}
$$

In general, any [square root of unity](https://en.wikipedia.org/wiki/Root_of_unity) can be used,

$$
A_n = \left(\cos(\theta) \pm i \sin(\theta) \right) \sqrt{\tfrac{2}{a}}
$$

where $k$ is any real number. The arbitrariness of the *phase* of the wavefunction is an important feature. Because the wavefunction can be imaginary (e.g., if you choose $A_n = i \sqrt{\tfrac{2}{a}}$), it is obvious that the wavefunction is not an observable property of a system. **The wavefunction is only a mathematical tool for quantum mechanics; it is not a physical object.**

Summarizing, the (normalized) wavefunction for a particle with mass $m$ confined to a one-dimensional box with length $a$ can be written as:

$$
\psi_n(x) = \sqrt{\tfrac{2}{a}} \sin\left(\tfrac{n \pi x}{a}\right) \qquad \qquad n=1,2,3,\ldots
$$

Note that in this case, the normalization condition is the same for all $n$; that is an unusual property of the particle-in-a-box wavefunction.

While this normalization convention is used 99% of the time, there are some cases where it is more convenient to make a different choice for the amplitude of the wavefunctions. I say this to remind you that normalization the wavefunction is something we do for convenience; it is not required by physics!

## Normalization Check
One advantage of using Jupyter is that we can easily check our (symbolic) mathematics. Let's confirm that the wavefunction is normalized by evaluating,

$$
\int_0^a \left| \psi_n(x) \right|^2 \, dx
$$

```python
# Execute this code block to import required objects.

# Note: The numpy library from autograd is imported, which behaves the same as
#       importing numpy directly. However, to make automatic differentiation work,
#       you should NOT import numpy directly by `import numpy as np`.

import autograd.numpy as np
from autograd import elementwise_grad as egrad

# import numpy as np
from scipy.integrate import trapezoid, quad
from scipy import constants

import ipywidgets as widgets

import matplotlib.pyplot as plt

# set the size of the plot
# plt.rcParams['figure.figsize'] = [10, 5]
```

```python
# Define a function for the wavefunction
def compute_wavefunction(x, n, a):
    """Compute 1-dimensional particle-in-a-box wave-function value(s).

    Parameters
    ----------
    x: float or np.ndarray
        Position of the particle.
    n: int
        Quantum number value.
    a: float
        Length of the box.
    """
    # check argument n
    if not (isinstance(n, int) and n > 0):
        raise ValueError("Argument n should be a positive integer.")
    # check argument a
    if a <= 0.0:
        raise ValueError("Argument a should be positive.")
    # check argument x
    if not (isinstance(x, float) or hasattr(x, "__iter__")):
        raise ValueError("Argument x should be a float or an array!")

    # compute wave-function
    value = np.sqrt(2 / a) * np.sin(n * np.pi * x / a)

    # set wave-function values out of the box equal to zero
    if hasattr(x, "__iter__"):
        value[x > a] = 0.0
        value[x < 0] = 0.0
    else:
        if x < 0.0 or x > a:
            value = 0.0
    return value

# Define a function for the wavefunction squared
def compute_probability(x, n, a):
    """Compute 1-dimensional particle-in-a-box probablity value(s).

    See `compute_wavefunction` parameters.
    """
    return compute_wavefunction(x, n, a)**2

#This next bit of code just prints out the normalization error
def check_normalization(a, n):
    #check the computed values of the moments against the analytic formula
    normalization,error = quad(compute_probability, 0, a, args=(n, a))
    print("Normalization of wavefunction = ", normalization)

#Principle quantum number:
n = 1

#Box length:
a = 1

check_normalization(a, n)
```

    Normalization of wavefunction =  1.0000000000000002

## The Energies of the Particle in a Box
How do we compute the energy of a particle in a box? All we need to do is substitute the eigenfunctions of the Hamiltonian, $\psi_n(x)$ back into the Schrödinger equation to determine the eigenenergies, $E_n$. That is, from

$$
\hat{H} \psi_n(x) = E_n \psi_n(x)
$$

we deduce

$$
\begin{align}
-\frac{\hbar^2}{2m} \frac{d^2}{dx^2} \left( A_n \sin \left( \frac{n \pi x}{a}\right) \right)
 &= E_n  \left( A_n \sin \left( \frac{n \pi x}{a}\right) \right) \\
 -A_n \frac{\hbar^2}{2m} \frac{d}{dx} \left( \frac{n \pi}{a} \cos \left( \frac{n \pi x}{a}\right) \right)
 &= E_n  \left( A_n \sin \left( \frac{n \pi x}{a}\right) \right) \\
  A_n \frac{\hbar^2}{2m} \left( \frac{n \pi}{a} \right)^2 \sin \left( \frac{n \pi x}{a}\right)
 &= E_n  \left( A_n \sin \left( \frac{n \pi x}{a}\right) \right) \\
   \frac{\hbar^2 n^2 \pi^2}{2ma^2}
 &= E_n
\end{align}
$$

Using the definition of $\hbar$, we can rearrange this to:

$$
\begin{align}
E_n &= \frac{\hbar^2 n^2 \pi^2}{2ma^2} \qquad \qquad n=1,2,3,\ldots\\
&= \frac{h^2 n^2}{8ma^2}
\end{align}
$$

Notice that only certain energies are allowed. This is a fundamental principle of quantum mechanics, and it is related to the "waviness" of particles. Certain "frequencies" are resonant, and other "frequencies" cannot be observed. *The **only** energies that can be observed for a particle-in-a-box are the ones given by the above formula.*

## Zero-Point Energy
Naïvely, you might expect that the lowest-energy state of a particle in a box has zero energy. (The potential in the box is zero, after all, so shouldn't the lowest-energy state be the state with zero kinetic energy? And if the kinetic energy were zero and the potential energy were zero, then the total energy would be zero.)

But this doesn't happen. It turns out that you can never "stop" a quantum particle; it always has a zero-point motion, typically a resonant oscillation about the lowest-potential-energy location(s). Indeed, the more you try to confine a particle to stop it, the bigger its kinetic energy becomes. This is clear in the particle-in-a-box, which has only kinetic energy. There the (kinetic) energy increases rapidly, as $a^{-2}$, as the box becomes smaller:

$$
T_n = E_n = \frac{h^2n^2}{8ma^2}
$$

The residual energy in the electronic ground state is called the **zero-point energy**,

$$
E_{\text{zero-point energy}} = \frac{h^2}{8ma^2}
$$

The existence of the zero-point energy, and the fact that zero-point kinetic energy is always positive, is a general feature of quantum mechanics.
> **Zero-Point Energy Principle:** Let $V(x)$ be a nonnegative potential. The ground-state energy is always greater than zero.

More generally, for any potential that is bound from below,

$$
V_{\text{min}}= \min_x V(x)
$$

the ground-state energy of the system satisfies $E_{\text{zero-point energy}} > V_{\text{min}}$.
>Nuance: There is a tiny mathematical footnote here; there are some $V(x)$ for which there are *no* bound states. In such cases, e.g., $V(x) = \text{constant}$, it is possible for $E = V_{\text{min}}$.)

## Atomic Units
Because Planck's constant and the electron mass are tiny numbers, it is often useful to use [atomic units](https://en.wikipedia.org/wiki/Hartree_atomic_units) when performing calculations. We'll learn more about atomic units later but, for now, we only need to know that, in atomic units, $\hbar$, the mass of the electron, $m_e$, the charge of the electron, $e$, and the average (mean) distance of an electron from the nucleus in the Hydrogen atom, $a_0$, are all defined to be equal to 1.0 in atomic units.

$$
\begin{align}
\hbar &= \frac{h}{2 \pi} = 1.0545718 \times 10^{-34} \text{ J s}= 1 \text{ a.u.} \\
m_e &= 9.10938356 \times 10^{-31} \text{ kg} = 1 \text{ a.u.} \\
e &= 1.602176634 \times 10^-19 \text{ C} = 1  \text{ a.u.} \\
a_0 &= 	5.291772109 \times 10^{-11} \text{ m} = 1 \text{ Bohr} = 1 \text{ a.u.}
\end{align}
$$

The unit of energy in atomic units is called the Hartree,

$$
E_h =4.359744722 \times 10^{-18} \text{ J} = 1 \text{ Hartree} = 1 \text{ a.u.}
$$

and the ground-state (zero-point) energy of the Hydrogen atom is $-\tfrac{1}{2} E_h$.

We can now define functions for the eigenenergies of the 1-dimensional particle in a box:

```python
# Define a function for the energy of a particle in a box
# with length a and quantum number n [in atomic units!]
# The length is input in Bohr (atomic units)
def compute_energy(n, a):
    "Compute 1-dimensional particle-in-a-box energy."
    return n**2 * np.pi**2 / (2 * a**2)

# Define a function for the energy of an electron in a box
# with length a and quantum number n [in SI units!].
# The length is input in meters.
def compute_energy_si(n, a):
    "Compute 1-dimensional particle-in-a-box energy."
    return n**2 * constants.h**2 / (8 * constants.m_e* a**2)

#Define variable for atomic unit of length in meters
a0 = constants.value('atomic unit of length')

#This next bit of code just prints out the energy in atomic and SI units
def print_energy(a, n):
    print(f'The energy of an electron in a box of length {a:.2f} a.u. with '
       f'quantum number {n} is {compute_energy(n, a):.2f} a.u..')
    print(f'The energy of an electron in a box of length {a*a0:.2e} m with '
       f'quantum number {n} is {compute_energy_si(n, a*a0):.2e} Joules.')

#Principle quantum number:
n = 1

#Box length:
a = 0.1

print_energy(a, n)
```

    The energy of an electron in a box of length 0.10 a.u. with quantum number 1 is 493.48 a.u..
    The energy of an electron in a box of length 5.29e-12 m with quantum number 1 is 2.15e-15 Joules.

#### 📝 Exercise: Write a function that returns the length, $a$, of a box for which the lowest-energy-excitation of the ground state, $n = 1 \rightarrow n=2$, corresponds to the system absorbing light with a given wavelength, $\lambda$. The input is $\lambda$; the output is $a$.

## Postulate: The wavefunction contains all the physically meaningful information about a system.
While the wavefunction is not itself observable, all observable properties of a system can be determined from the wavefunction. However, just because the wavefunction encapsulates all the *observable* properties of a system does not mean that it contains *all information* about a system. In quantum mechanics, some things are not observable. Consider that for the ground ($n=1$) state of the particle in a box, the root-mean-square average momentum,

$$
\bar{p}_{rms} = \sqrt{2m \cdot T} = \sqrt{(2m)\frac{h^2n^2}{8ma^2}} = \frac{hn}{2a}
$$

increases as you squeeze the box. That is, the more you try to constrain the particle in space, the faster it moves. You can't "stop" the particle no matter how hard you squeeze it, so it's impossible to exactly know where the particle is located. You can only determine its *average* position.

## Postulate: Observable Quantities Correspond to Linear, Hermitian Operators.
The *correspondence* principle says that for every classical observable there is a linear, Hermitian, operator that allows computation of the quantum-mechanical observable. An operator, $\hat{C}$ is linear if for any complex numbers $a$ and $b$, and any wavefunctions $\psi_1(x)$ and $\psi_2(x)$,

$$
\hat{C} \left(a \psi_1(x,t) + b \psi_2(x,t) \right) = a \hat{C} \psi_1(x,t) + b \hat{C} \psi_2(x,t)
$$

Similarly, an operator is Hermitian if it satisfies the relation,

$$
\int \psi_1^*(x,t) \hat{C} \psi_2(x,t) \, dx = \int \left( \hat{C}\psi_1(x,t) \right)^*  \psi_2(x,t) \, dx
$$

or, equivalently,

$$
\int \psi_1^*(x,t) \left( \hat{C} \psi_2(x,t) \right) \, dx = \int \psi_2(x,t)\left( \hat{C} \psi_1(x,t) \right)^* \, dx
$$

That is for a linear operator, the linear operator applied to a sum of wavefunctions is equal to the sum of the linear operators directly applied to the wavefunctions separately, and the linear operator applied to a constant times a wavefunction is the constant times the linear operator applied directly to the wavefunction. A Hermitian operator can apply forward (towards $\psi_2(x,t)$) or backwards (towards $\psi_1(x,t)$). This is very useful, because sometimes it is much easier to apply an operator in one direction.

We've already been introduced to the quantum-mechanical operators for the momentum,

$$
\hat{p} = -i \hbar \tfrac{d}{dx}
$$

and the kinetic energy,

$$
\hat{T} = -\tfrac{\hbar^2}{2m} \tfrac{d^2}{dx^2}
$$

These operators are linear because the derivative of a sum is the sum of the derivatives, and the derivative of a constant times a function is that constant times the derivative of the function. These operators are also Hermitian. For example, to show that the momentum operator is Hermitian:

$$
\begin{align}
\int_{-\infty}^{\infty} \psi_1^*(x,t) \hat{p} \psi_2(x,t) dx &= \int_{-\infty}^{\infty} \psi_1^*(x,t) \left( -i \hbar \tfrac{d}{dx} \right) \psi_2(x,t) dx \\
&= -i \hbar \int_{-\infty}^{\infty} \tfrac{d}{dx} \left(\psi_1^*(x,t)\psi_2(x,t)  \right) - \left(
 \psi_2(x,t) \tfrac{d}{dx} \psi_1^*(x,t)\right)  dx
\end{align}
$$

Here we used the product rule for derivatives, $f(x)\tfrac{dg}{dx} = \tfrac{d f(x) g(x)}{dx} - g(x) \tfrac{df}{dx}$. Using the fundamental theorem of calculus and the fact the probability of observing a particle at $\pm \infty$ is zero, and therefore the wavefunctions at infinity are also zero, one knows that

$$
\int_{-\infty}^{\infty} \tfrac{d}{dx} \left(\psi_1^*(x,t)\psi_2(x,t)  \right) = \left[  \psi_1^*(x,t)\psi_2(x,t)\right]_{-\infty}^{\infty} = 0
$$

Therefore the above equation can be simplified to

$$
\begin{align}
\int_{-\infty}^{\infty} \psi_1^*(x,t) \hat{p} \psi_2(x,t) dx &=i \hbar \int_{-\infty}^{\infty}   \psi_2(x,t) \tfrac{d}{dx} \psi_1^*(x,t)  dx \\
&= \int_{-\infty}^{\infty}   \psi_2(x,t) i \hbar  \tfrac{d}{dx} \psi_1^*(x,t) dx \\
&= \int_{-\infty}^{\infty}  \psi_2(x,t) \left( -i \hbar  \tfrac{d}{dx} \psi_1^*(x,t)\right)  dx \\
&= \int_{-\infty}^{\infty}  \psi_2(x,t) \left( \hat{p} \psi_1(x,t)\right)^*  dx
\end{align}
$$

The expectation value of the momentum of a particle-in-a-box is always zero. This is intuitive, since electrons (on average) are neither moving to the right nor to the left inside the box: if they were, then the box itself would need to be moving. Indeed, for any real wavefunction, the average momentum is always zero. This follows directly from the previous derivation with $\psi_1^*(x,t) = \psi_2(x,t)$. Thus:

$$
\begin{align}
\int_{-\infty}^{\infty} \psi_2(x,t) \hat{p} \psi_2(x,t) dx
&=i \hbar \int_{-\infty}^{\infty}   \psi_2(x,t) \tfrac{d}{dx} \psi_2(x,t)  dx \\
&=-i \hbar \int_{-\infty}^{\infty}  \psi_2(x,t) \left( \tfrac{d}{dx} \psi_2(x,t)\right)  dx \\
&= 0
\end{align}
$$

The last line follows because the only number that is equal to its negative is zero. (That is, $x=-x$ if and only if $x=0$.) It is a subtle feature that the eigenfunctions of a real-valued Hamiltonian operator can always be chosen to be real-valued themselves, so their average momentum is clearly zero. We often denote quantum-mechanical expectation values with the shorthand,

$$
\langle \hat{p} \rangle =0
$$

The momentum-squared of the particle-in-a-box is easily computed from the kinetic energy,

$$
\langle \hat{p}^2 \rangle = \int_0^a \psi_n(x) \hat{p}^2 \psi_n(x) dx = \int_0^a \psi_n(x) \left(2m\hat{T}\right) \psi_n(x) dx = 2m E_n = \frac{h^2n^2}{4a^2}
$$

Intuitively, since the box is symmetric about $x=\tfrac{a}{2}$, the particle has equal probability of being in the first-half and the second-half of the box. So the average position is expected to be

$$
\langle x \rangle =\tfrac{a}{2}
$$

We can confirm this by explicit integration,

$$
\begin{align}
\langle x \rangle &= \int_0^a \psi_n^*(x)\, x \,\psi_n(x) dx \\
&= \int_0^a \left(\sqrt{\tfrac{2}{a}} \sin\left(\tfrac{n \pi x}{a} \right)\right) x \left(\sqrt{\tfrac{2}{a}}\sin\left(\tfrac{n \pi x}{a} \right)\right) dx \\
&= \tfrac{2}{a} \int_0^a  x \sin^2\left(\tfrac{n \pi x}{a} \right) dx \\
&= \tfrac{2}{a} \left[ \tfrac{x^2}{4} - x \tfrac{\sin \tfrac{2n \pi x}{a}}{\tfrac{4 n \pi}{a}}
- \tfrac{\cos \tfrac{2n \pi x}{a}}{\tfrac{8 n^2 \pi^2}{a^2}}
 \right]_0^a \\
 &= \tfrac{2}{a} \left[ \tfrac{a^2}{4} - 0 - 0 \right] \\
 &= \tfrac{a}{2}
\end{align}
$$

Similarly, we expect that the expectation value of $\langle x^2 \rangle$ will be proportional to $a^2$. We can confirm this by explicit integration,

$$
\begin{align}
\langle x^2 \rangle &= \int_0^a \psi_n^*(x)\, x^2 \,\psi_n(x) dx \\
&= \int_0^a \left(\sqrt{\tfrac{2}{a}} \sin\left(\tfrac{n \pi x}{a} \right)\right) x^2 \left(\sqrt{\tfrac{2}{a}}\sin\left(\tfrac{n \pi x}{a} \right)\right) dx \\
&= \tfrac{2}{a} \int_0^a  x^2 \sin^2\left(\tfrac{n \pi x}{a} \right) dx \\
&= \tfrac{2}{a} \left[ \tfrac{x^3}{6}
 - x^2 \tfrac{\sin \tfrac{2n \pi x}{a}}{\tfrac{4 n \pi}{a}}
 - x \tfrac{\cos \tfrac{2n \pi x}{a}}{\tfrac{4 n^2 \pi^2}{a^2}}
 - \tfrac{\sin \tfrac{2n \pi x}{a}}{\tfrac{8 n^3 \pi^3}{a^3}}
 \right]_0^a \\
 &= \tfrac{2}{a} \left[ \tfrac{a^3}{6} - 0 - \tfrac{a}{{\tfrac{4 n^2 \pi^2}{a^2}}} - 0 \right] \\
  &= \tfrac{2}{a} \left[ \tfrac{a^3}{6} - \tfrac{a^3}{4 n^2 \pi^2} \right] \\
  &= a^2\left[ \tfrac{1}{3} - \tfrac{1}{2 n^2 \pi^2} \right]
\end{align}
$$

We can verify these formulas by explicit integration.

```python
#Compute <x^power>, the expectation value of x^power
def compute_moment(x, n, a, power):
    """Compute the x^power moment of the 1-dimensional particle-in-a-box.

    See `compute_wavefunction` parameters.
    """
    return compute_probability(x, n, a)*x**power

#This next bit of code just prints out the values.
def check_moments(a, n):
    #check the computed values of the moments against the analytic formula
    avg_r,error = quad(compute_moment, 0, a, args=(n, a, 1))
    avg_r2,error = quad(compute_moment, 0, a, args=(n, a, 2))
    print(f"<r> computed = {avg_r:.5f}")
    print(f"<r> analytic = {a/2:.5f}")
    print(f"<r^2> computed = {avg_r2:.5f}")
    print(f"<r^2> analytic = {a**2*(1/3 - 1./(2*n**2*np.pi**2)):.5f}")

#Principle quantum number:
n = 1

#Box length:
a = 1

check_moments(a, n)
```

    <r> computed = 0.50000
    <r> analytic = 0.50000
    <r^2> computed = 0.28267
    <r^2> analytic = 0.28267

## Heisenberg Uncertainty Principle
The previous example gives a first example of the more general [Heisenberg Uncertainty Principle](https://en.wikipedia.org/wiki/Uncertainty_principle). One specific manifestation of the Heisenberg Uncertaity Principle is that the variance of the position, $\sigma_x^2 = \langle x^2 \rangle - \langle x \rangle^2$, times the variance of the momentum. $\sigma_p^2 = \langle \hat{p}^2 \rangle - \langle \hat{p} \rangle^2$ is greater than $\tfrac{\hbar^2}{4}$. We can verify this formula for the particle in a box.

$$
\begin{align}
\tfrac{\hbar^2}{4} &\le \sigma_x^2 \sigma_p^2 \\
&= \left( a^2\left[ \tfrac{1}{3} - \tfrac{1}{2 n^2 \pi^2} \right] - \tfrac{a^2}{4} \right)
 \left( \frac{h^2n^2}{4a^2} - 0 \right) \\
&= \left( a^2\left[ \tfrac{1}{3} - \tfrac{1}{2 n^2 \pi^2} \right] - \tfrac{a^2}{4} \right)
 \left( \frac{\hbar^2 \pi^2 n^2}{a^2} \right) \\
&= \hbar^2 \pi^2 n^2 \left(\tfrac{1}{12} - \tfrac{1}{2 n^2 \pi^2} \right)
\end{align}
$$

The right-hand-side gets larger and larger as $n$ increases, so the largest value occurs where:

$$
\begin{align}
\tfrac{\hbar^2}{4} &\le \hbar^2 \pi^2 \left(\tfrac{1}{12} - \tfrac{1}{2 \pi^2} \right) \\
&= 0.32247 \hbar^2
\end{align}
$$

## Double-Checking the Energy of a Particle-in-a-Box
To check the energy of the particle in a box, we can compute the kinetic energy density, then integrate it over all space. That is, we define:

$$
\tau_n(x) = \psi_n^*(x) \left(-\tfrac{\hbar^2}{2m}\tfrac{d^2}{dx^2}\right) \psi_n(x)
$$

and then the kinetic energy (which is the energy for the particle in a box) is

$$
T_n = \int_0^a \tau_n(x) dx
$$

> *Note:* In fact there are several different equivalent definitions of the kinetic energy density, but this is not very important in an introductory quantum chemistry course. All of the kinetic energy densities give the same total kinetic energy. However, because the kinetic energy density, $\tau(x)$, represents the kinetic energy of a particle at the point $x$, and it is impossible to know the momentum (or the momentum-squared, ergo the kinetic energy) exactly at a given point in space according to the Heisenberg uncertainty principle), there can be no unique definition for $\tau(x)$.

```python
#The next few lines just set up the sliders for setting parameters.
#Principle quantum number slider:

def compute_wavefunction_derivative(x, n, a, order=1):
    """Compute 1-dimensional particle-in-a-box kinetic energy density.
    """
    if not (isinstance(order, int) and n > 0):
        raise ValueError("Argument order is expected to be a positive integer!")

    def wavefunction(x):
        v = np.sqrt(2 / a) * np.sin(n * np.pi * x / a)
        return v

    # compute derivative
    deriv = egrad(wavefunction)
    for _ in range(order - 1):
        deriv = egrad(deriv)

    # return zero for x values out of the box
    deriv = deriv(x)
#     deriv[x < 0] = 0.0
#     deriv[x > a] = 0.0

    if hasattr(x, "__iter__"):
        deriv[x > a] = 0.0
        deriv[x < 0] = 0.0
    else:
        if x < 0.0 or x > a:
            deriv = 0.0

    return deriv

def compute_kinetic_energy_density(x, n, a):
    """Compute 1-dimensional particle-in-a-box kinetic energy density.

    See `compute_wavefunction` parameters.
    """
    # evaluate wave-function and its 2nd derivative w.r.t. x
    wf = compute_wavefunction(x, n, a)
    d2 = compute_wavefunction_derivative(x, n, a, order=2)
    return -0.5 * wf * d2

#This next bit of code just prints out the values.
def check_energy(a, n):
    #check the computed values of the moments against the analytic formula
    ke,error = quad(compute_kinetic_energy_density, 0, a, args=(n, a))
    energy = compute_energy(n, a)
    print(f"The energy computed by integrating the k.e. density is {ke:.5f}")
    print(f"The energy computed directly is {energy:.5f}")

#Principle quantum number:
n = 1

#Box length:
a = 17

check_energy(a, n)
```

    The energy computed by integrating the k.e. density is 0.01708
    The energy computed directly is 0.01708

## Visualizing the Particle-in-a-Box Wavefunctions, Probabilities, etc.
In the next code block, the wavefunction, probability density, derivative, second-derivative, and kinetic energy density for the particle-in-a-box are shown. Notice that the kinetic-energy-density is proportional to the probability density, and that the first and second derivatives are not zero at the edge of the box, but the wavefunction and probability density are. It's useful to change the parameters in the below figures to build your intuition for the particle-in-a-box.

```python
#This next bit of code makes the plots and prints out the energy
def make_plots(a, n):
    #check the computed values of the moments against the analytic formula
    energy = compute_energy(n, a)
    print(f"The energy computed directly is {energy:.5f}")
    # sample x coordinates
    x = np.arange(-0.6, a + 0.6, 0.01)

    # evaluate wave-function & probability
    wf = compute_wavefunction(x, n, a)
    pr = compute_probability(x, n, a)

    # evaluate 1st & 2nd derivative of wavefunction w.r.t. x
    d1 = compute_wavefunction_derivative(x, n, a, order=1)
    d2 = compute_wavefunction_derivative(x, n, a, order=2)

    # evaluate kinetic energy density
    kin = compute_kinetic_energy_density(x, n, a)
    #print("Integrate KED = ", trapezoid(kin, x))

    # set the size of the plot
    plt.rcParams['figure.figsize'] = [15, 10]
    plt.rcParams['font.family'] = 'DejaVu Sans'
    plt.rcParams['font.serif'] = ['Times New Roman']
    plt.rcParams['mathtext.fontset'] = 'stix'
    plt.rcParams['xtick.labelsize'] = 15
    plt.rcParams['ytick.labelsize'] = 15

    # define four subplots
    fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2)
    fig.suptitle(f'a={a}  n={n}  E={compute_energy(n, a):.4f} a.u.', fontsize=35, fontweight='bold')

    # plot 1
    ax1.plot(x, wf, linestyle='--', label=r'$\psi(x)$', linewidth=3)
    ax1.plot(x, pr, linestyle='-', label=r'$\|\psi(x)\|^2$', linewidth=3)
    ax1.legend(frameon=False, fontsize=20)
    ax1.set_xlabel('x coordinates', fontsize=20)

    # plot 2
    ax2.plot(x, d1, linewidth=3, c='k')
    ax2.set_xlabel('x coordinates', fontsize=20)
    ax2.set_ylabel(r'$\frac{d\psi(x)}{dx}$', fontsize=30, rotation=0, labelpad=25)

    # plot 3
    ax3.plot(x, d2, linewidth=3, c='g', )
    ax3.set_xlabel('x coordinates', fontsize=20)
    ax3.set_ylabel(r'$\frac{d^2\psi(x)}{dx^2}$', fontsize=25, rotation=0, labelpad=25)

    # plot 4
    ax4.plot(x, kin, linewidth=3, c='r')
    ax4.set_xlabel('x coordinates', fontsize=20)
    ax4.set_ylabel('Kinetic Energy Density', fontsize=16)

    # adjust spacing between plots
    plt.subplots_adjust(left=0.125,
                    bottom=0.1,
                    right=0.9,
                    top=0.9,
                    wspace=0.35,
                    hspace=0.35)
    #Show Plot
    plt.show()

#Principle quantum number slider:
n = 1

#Box length slider:
a = 1

make_plots(a, n)
```

## 🪞 Self-Reflection
- Can you think of other physical or chemical systems where the particle-in-a-box Hamiltonian would be appropriate?
- Can you think of another property density, besides the kinetic energy density, that cannot be uniquely defined in quantum mechanics?

## 🤔 Thought-Provoking Questions
- How would the wavefunction and ground-state energy change if you made a small change in the particle-in-a-box Hamiltonian, so that the right-hand-side of the box was a little higher than the left-hand-side?
- Any system with a zero-point energy of zero is classical. Why?
- How would you compute the probability of observing an electron at the center of a box if the box contained 2 electrons? If it contained 4 electrons? If it contained 8 electrons? The probability of observing an electron at the center of a box with 3 electrons is sometimes *lower* than the probability of observing an electron at the center of a box with 2 electrons. Why?
- Demonstrate that the kinetic energy operator is linear and Hermitian.
- What is the lowest-energy excitation energy for the particle-in-a-box?
- Suppose you wanted to design a one-dimensional box containing a single electron that absorbed blue light? How long would the box be?

## ❓ Knowledge Tests
- Questions about the Particle-in-a-Box and related concepts. [GitHub Classroom Link](https://classroom.github.com/a/1Y48deKP)

## 👩🏽💻 Assignments
- Compute and understand expectation values by computing moments of the particle-in-a-box [assignment](https://github.com/McMasterQM/PinBox-Moments/blob/main/moments.ipynb). [Github classroom link]https://classroom.github.com/a/9yzWI5Vt.
- This [assignment](https://github.com/McMasterQM/Sudden-Approximation/blob/main/SuddenPinBox.ipynb) on the sudden approximation provides an introduction to time-dependent phenomena. [Github classroom link](https://classroom.github.com/a/yBzABlb-).

## 🔁 Recapitulation
- Write the Hamiltonian, time-independent Schrödinger equation, eigenfunctions, and eigenvalues for the one-dimensional particle in a box.
- Play around with the eigenfunctions and eigenenergies of the particle-in-a-box to build intuition for them.
- How would you compute the uncertainty in $x^4$?
- Practice your calculus by explicitly computing, using integration by parts, $\langle x \rangle$ and $\langle x^2 \rangle$. This can be implemented in a [Jupyter notebook](x4_mocked.ipynb).

## 🔮 Next Up...
- Postulates of Quantum Mechanics
- Multielectron particle-in-a-box.
- Multidimensional particle-in-a-box.
- Harmonic Oscillator

## 📚 References
My favorite sources for this material are:
- [Randy's book](https://github.com/PaulWAyers/IntroQChem/blob/main/documents/DumontBook.pdf) has an excellent treatment of the particle-in-a-box model, including several extensions to the material covered here. (Chapter 3)
- Also see my (pdf) class [notes](https://github.com/PaulWAyers/IntroQChem/blob/main/documents/PinBox.pdf).
- Also see my notes on the [mathematical structure of quantum mechanics](https://github.com/PaulWAyers/IntroQChem/blob/main/documents/LinAlgAnalogy.pdf).
- [Davit Potoyan's](https://www.chem.iastate.edu/people/davit-potoyan) Jupyter-book covers the particle-in-a-box in [chapter 4](https://dpotoyan.github.io/Chem324/intro.html) are especially relevant here.
- D. A. MacQuarrie, Quantum Chemistry (University Science Books, Mill Valley California, 1983)
- [An excellent explanation of the link to the spectrum of cyanine dyes](https://pubs.acs.org/doi/10.1021/ed084p1840)
- Chemistry Libre Text: [one dimensional](https://chem.libretexts.org/Bookshelves/Physical_and_Theoretical_Chemistry_Textbook_Maps/Map%3A_Physical_Chemistry_for_the_Biosciences_(Chang)/11%3A_Quantum_Mechanics_and_Atomic_Structure/11.08%3A_Particle_in_a_One-Dimensional_Box)
and [multi-dimensional](https://chem.libretexts.org/Bookshelves/Physical_and_Theoretical_Chemistry_Textbook_Maps/Supplemental_Modules_(Physical_and_Theoretical_Chemistry)/Quantum_Mechanics/05.5%3A_Particle_in_Boxes/Particle_in_a_3-Dimensional_box)
- [McQuarrie and Simon summary](https://chem.libretexts.org/Bookshelves/Physical_and_Theoretical_Chemistry_Textbook_Maps/Map%3A_Physical_Chemistry_(McQuarrie_and_Simon)/03%3A_The_Schrodinger_Equation_and_a_Particle_in_a_Box)

There are also some excellent wikipedia articles:
- [Particle in a Box](https://en.wikipedia.org/wiki/Particle_in_a_box)
- [Particle on a Ring](https://en.wikipedia.org/wiki/Particle_in_a_ring)
- [Postulates of Quantum Mechanics](https://en.wikipedia.org/wiki/Mathematical_formulation_of_quantum_mechanics#Postulates_of_quantum_mechanics)

```python

```


```
# default_exp definition.interval
```

# definition.interval

```
#hide
from mathbook.utility.markdown import *
from mathbook.configs import *
# uncomment for editing.
# DESTINATION = 'notebook'
# ORIGIN = 'notebook'
```

## Interval

```
#export
if __name__ == '__main__':
    embed_markdown_file('definition.interval.md',
                        destination=DESTINATION, origin=ORIGIN)
```

    [(65, 92)]

For real numbers $a,b$, an **interval** between $a$ and $b$ is a [subset](https://hyunjongkimmath.github.io/mathbook/definition.subset.html) of $\mathbb{R}$[^1] containing all of the real numbers between $a$ and $b$. An interval is of one of the following forms, depending on which of the two endpoints (i.e., $a$ and $b$) it contains:

1. The **open interval**, denoted $(a,b)$, is the set of real numbers containing all real numbers between $a$ and $b$, not including $a$ and $b$:
$$
\begin{align}
(a,b) := \{x \in \mathbb{R}: a < x < b \}
\end{align}
$$
2. The **closed interval**, denoted $[a,b]$, is the set of real numbers containing all real numbers between $a$ and $b$, including both $a$ and $b$:
$$
\begin{align}
[a,b] := \{x \in \mathbb{R}: a \leq x \leq b \}
\end{align}
$$
3. The **half open intervals**, denoted $(a,b]$ and $[a,b)$, are the sets of real numbers containing all real numbers between $a$ and $b$, including one of $a$ and $b$ and not the other:
$$
\begin{align}
(a,b] &:= \{x \in \mathbb{R}: a < x \leq b \} \\
[a,b) &:= \{x \in \mathbb{R}: a \leq x < b \}
\end{align}
$$

[^1]:https://hyunjongkimmath.github.io/mathbook/notation.basic.html#$\mathbb{R}$


# Fundamentals of Data Science
Winter Semester 2021

## Prof. Fabio Galasso, Guido D'Amely, Alessandro Flaborea, Luca Franco, Muhammad Rameez Ur Rahman and Alessio Sampieri
<galasso@di.uniroma1.it>, <damely@di.uniroma1.it>, <flaborea@di.uniroma1.it>, <franco@diag.uniroma1.it>, <rahman@di.uniroma1.it>, <alessiosampieri27@gmail.com>

## Exercise 2: Classification

In Exercise 2, you will re-derive and implement logistic regression and optimize the parameters with Gradient Descent and with the Newton's method. Also, in this exercise you will re-derive and implement Gassian Discriminant Analysis.
We will use datasets generated from the make_classification function from the SkLearn library. Its first output contains the feature values $x^{(i)}_1$ and $x^{(i)}_2$ for the $i$-th data sample $x^{(i)}$. The second contains the ground truth label $y^{(i)}$ for each corresponding data sample.

The completed exercise should be handed in as a single notebook file. Use Markdown to provide equations. Use the code sections to provide your scripts and the corresponding plots.
Submit it by sending an email to galasso@di.uniroma1.it, flaborea@di.uniroma1.it, franco@diag.uniroma1.it and alessiosampieri27@gmail.com  by Wednesday November 17th 2021, 23:59.

## Notation

- $x^i$ is the $i^{th}$ feature vector
- $y^i$ is the expected outcome for the $i^{th}$ training example
- $m$ is the number of training examples
- $n$ is the number of features

Let's start by setting up our Python environment and importing the required libraries:

```python
%matplotlib inline
import numpy as np # imports a fast numerical programming library
import scipy as sp # imports stats functions, amongst other things
import matplotlib as mpl # this actually imports matplotlib
import matplotlib.cm as cm # allows us easy access to colormaps
import matplotlib.pyplot as plt # sets up plotting under plt
import pandas as pd # lets us handle data as dataframes
from sklearn.datasets import make_classification
import seaborn as sns

# sets up pandas table display
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)

import seaborn as sns # sets up styles and gives us more plotting options
```

## Question 1: Logistic Regression with Gradient Ascent **(10 Points)**

### Code and Theory

#### Exercise 1.a **(3 Points)** Equations for the log likelihood, its gradient, and the gradient ascent update rule.

Write and simplify the likelihood $L(\theta)$ and log-likelihood $l(\theta)$ of the parameters $\theta$.

Recall the probabilistic interpretation of the hypothesis $h_\theta(x)= P(y=1|x;\theta)$ and that $h_\theta(x)=\frac{1}{1+\exp(-\theta^T x)}$.

Also derive the gradient $\frac{\delta l(\theta)}{\delta \theta_j}$ of $l(\theta)$ and write the gradient update equation.

Question: Are we looking for a local minimum or a local maximum using the gradient ascent rule?

################# Do not write above this line #################

- $P(Y=1|x; \theta) = h_{\theta}(x)$

- $P(Y=0|x; \theta) = 1 - h_{\theta}(x)$

Given these two points, we can say that in general we have:

$P(y|x;\theta)=h_{\theta}(x)^{y} (1+h_{\theta}(x))^{1-y}$

We must observe that for $y=1$, we get only $h_{\theta}(x)^{y}$, instead for $y=0$, we just get $(1+h_{\theta}(x))^{1-y}$.

The *Likelihood $L(\theta)$* is given by:

$$L(\theta) = P(\vec{y}|x; \theta)  \\
 = \prod_{i=1}^{m} P(y^{(i)}|x^{(i)};\theta)  \\
 = \prod_{i=1}^{m} h_{\theta}(x^{(i)})^{y^{(i)}}(1-h_{\theta}(x^{(i)}))^{1-y^{(i)}}  \\
 = \prod_{i=1}^{m} \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)^{y^{(i)}}\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)^{1-y^{(i)}}
$$

Instead, in order to find the *Log-likelihood $l(\theta)$*, we must apply the natural logarithm function to the likelihood function previously found, and we get:

$$
l(\theta) = \log L(\theta)
 = \sum_{i=1}^{m} y^{(i)}\log h_{\theta}(x^{(i)})+(1-y^{(i)})\log(1-h_{\theta}(x^{(i)}))  \\
 = \sum_{i=1}^{m} y^{(i)}\log \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg) + (1-y^{(i)})\log\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)
$$

We proceed deriving the gradient $\frac{\delta l(\theta)}{\delta \theta_j}$ of $l(\theta)$ as follows:
$$
\frac{\delta l(\theta)}{\delta \theta_j} = \frac{\delta}{\delta \theta_j} \Bigg(\sum_{i=1}^{m} y^{(i)}\log \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg) + (1-y^{(i)})\log\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)\Bigg)\\
= \Bigg[y^{(i)}-\bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)\Bigg]x^{(i)}
$$

The procedure of the _gradient ascent rule_ leads us to a local maximum of our (differentiable) function.

################# Do not write below this line #################

#### Exercise 1.b **(7 Points)** Implementation of logistic regression with Gradient Ascent

Code up the equations above to learn the logistic regression parameters. The dataset used here is created using the make_classification function present in the SkLearn library. $x^{(i)}_1$ and $x^{(i)}_2$ represent the two features for the $i$-th data sample $x^{(i)}$ and $y^{(i)}$ is its ground truth label.

```python
X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=2, random_state=5)
X.shape, y.shape
```

    ((500, 2), (500,))

```python
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y);
```

Adding a column of 1's to $X$ to take into account the zero intercept

```python
x = np.hstack([np.ones((X.shape[0], 1)), X])
```

```python
[x[:5,:],x[-5:,:]] # Plot the first and last 5 lines of x, now containing features x0 (constant=1), x1 and x2
```

    [array([[ 1.        ,  2.25698215, -1.34710915],
            [ 1.        ,  1.43699308,  1.28420453],
            [ 1.        ,  0.57927295,  0.23690172],
            [ 1.        ,  0.42538132, -0.24611145],
            [ 1.        ,  1.13485101, -0.61162683]]),
     array([[ 1.        ,  1.56638944,  0.81749944],
            [ 1.        , -1.94913831, -1.90601147],
            [ 1.        ,  1.53440506, -0.11687238],
            [ 1.        , -0.39243599,  1.39209018],
            [ 1.        , -0.11881249,  0.96973739]])]

```python
[y[:5],y[-5:]] # Plot the first and last 5 lines of y
```

    [array([1, 1, 1, 0, 1]), array([1, 0, 0, 0, 1])]

Define the sigmoid function "sigmoid", the function to compute the gradient of the log likelihood  "grad_l" and the gradient ascent algorithm.

################# Do not write above this line #################

```python
def sigmoid(x):
    '''
    Function to compute the sigmoid of a given input x.

    Input:
    x: it's the input data matrix. The shape is (N, H)

    Output:
    g: The sigmoid of the input x
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return g

def log_likelihood(theta,features,target):
    '''
    Function to compute the log likehood of theta according to data x and label y

    Input:
    theta: it's the model parameter matrix.
    features: it's the input data matrix. The shape is (N, H)
    target: the label array

    Output:
    log_g: the log likehood of theta according to data x and label y
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################
    h_tt = (1/(1+exp(-theta*features)))
    log_l = sum(target*log(h_tt))+(1-target)*(1-log(h_tt))
    return log_l

def predictions(features, theta):
    '''
    Function to compute the predictions for the input features

    Input:
    theta: it's the model parameter matrix.
    features: it's the input data matrix. The shape is (N, H)

    Output:
    preds: the predictions of the input features
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return preds

def update_theta(theta, target, preds, features, lr):
    '''
    Function to compute the gradient of the log likelihood
    and then return the updated weights

    Input:
    theta: the model parameter matrix.
    target: the label array
    preds: the predictions of the input features
    features: it's the input data matrix. The shape is (N, H)
    lr: the learning rate

    Output:
    theta: the updated model parameter matrix.
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################
    return theta

def gradient_ascent(theta, features, target, lr, num_steps):
    '''
    Function to execute the gradient ascent algorithm

    Input:
    theta: the model parameter matrix.
    target: the label array
    num_steps: the number of iterations
    features: the input data matrix. The shape is (N, H)
    lr: the learning rate

    Output:
    theta: the final model parameter matrix.
    log_likelihood_history: the values of the log likelihood during the process
    '''

    log_likelihood_history = np.zeros(num_steps)

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return theta, log_likelihood_history

```

################# Do not write below this line #################

Check your grad_l implementation:
grad_l applied to the theta_test (defined below) should provide a value for log_l_test close to the target_value (defined below); in other words the error_test should be 0, up to machine error precision.

```python
target_value = -1.630501731599431

output_test  = log_likelihood(np.array([-7,4,1]),x,y)
error_test=np.abs(output_test-target_value)

print("{:f}".format(error_test))
```

    0.000000

Let's now apply the function gradient_ascent and print the final theta as well as theta_history

```python
# Initialize theta0
theta0 = np.zeros(x.shape[1])

# Run Gradient Ascent method
n_iter=1000
theta_final, log_l_history = gradient_ascent(theta0,x,y,lr=0.5,num_steps=n_iter)
print(theta_final)

```

    [-0.46097042  2.90036399  0.23146846]

Let's plot the log likelihood over iterations

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)),log_l_history,'b.')
```

Plot the data and the decision boundary:

```python
# Generate vector to plot decision boundary
x1_vec = np.linspace(X[:,0].min(),X[:,1].max(),2)

# Plot raw data
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X)

# Plot decision boundary
plt.plot(x1_vec,(-x1_vec*theta_final[1]-theta_final[0])/theta_final[2], color="red")
plt.ylim(X[:,1].min()-1,X[:,1].max()+1)
# Save the theta_final value for later comparisons
theta_GA = theta_final.copy()
```

################# Do not write above this line #################

Discuss these two points:
1. You have implemented the gradient ascent rule. Could we have also used gradient descent instead for the proposed problem? Why/Why not?
2. Let's deeply analyze how the learning rate $\alpha$ and the number of iterations affect the final results. Run the algorithm you have written for different values of $\alpha$ and the number of iterations and look at the outputs you get. Is the decision boundary influenced by these parameters change? Why do you think these parameters are affecting/not affecting the results?

################# Do not write below this line #################

## Question 2: Logistic Regression with non linear boundaries (7 points)

#### Exercise 2.a **(4 Points)** Polynomial features for logistic regression

Define new features, e.g. of 2nd and 3rd degrees, and learn a logistic regression classifier by using the new features, by using the gradient ascent optimization algorithm you defined in Question 1.

In particular, we would consider a polynomial boundary with equation:

$f(x_1, x_2) = c_0 + c_1 x_1 + c_2 x_2 + c_3 x_1^2 + c_4 x_2^2 + c_5 x_1 x_2 + c_6 x_1^3 + c_7 x_2^3 + c_8 x_1^2 x_2 + c_9 x_1 x_2^2$

We would therefore compute 7 new features: 3 new ones for the quadratic terms and 4 new ones for the cubic terms.

Create new arrays by stacking x and the new 7 features (in the order x1x1, x2x2, x1x2, x1x1x1, x2x2x2, x1x1x2, x1x2x2). In particular create x_new_quad by additionally stacking with x the quadratic features, and x_new_cubic by additionally stacking with x the quadratic and the cubic features.

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=2, random_state=5)
X.shape, y.shape
```

    ((500, 2), (500,))

```python
x = np.hstack([np.ones((X.shape[0], 1)), X])
```

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y);
```

```python
# First extract features x1 and x2 from x and reshape them to x1 vector arrays
x1 = x[:,1]
x2 = x[:,2]
x1 = x1.reshape(x1.shape[0], 1)
x2 = x2.reshape(x2.shape[0], 1)
print(x[:5,:]) # For visualization of the first 5 values
print(x1[:5,:]) # For visualization of the first 5 values
print(x2[:5,:]) # For visualization of the first 5 values
```

    [[ 1.          2.25698215 -1.34710915]
     [ 1.          1.43699308  1.28420453]
     [ 1.          0.57927295  0.23690172]
     [ 1.          0.42538132 -0.24611145]
     [ 1.          1.13485101 -0.61162683]]
    [[2.25698215]
     [1.43699308]
     [0.57927295]
     [0.42538132]
     [1.13485101]]
    [[-1.34710915]
     [ 1.28420453]
     [ 0.23690172]
     [-0.24611145]
     [-0.61162683]]

################# Do not write above this line #################

Your code here

```python

def new_features(x, degree=2):
    '''
    Function to create n-degree features from the input

    Input:
    x: the initial features
    degree: the maximum degree you wantthe features

    Output:
    features: the final features.
              2nd degree features must have the order [x, x1x1, x1x2, x2x2]
              2nd degree features must have the order [x, x1x1, x1x2, x2x2, x1x1x1, x1x1x2, x1x2x2, x2x2x2]
    '''

    features = np.ones(x[:,1].shape[0])

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return features

```

################# Do not write below this line #################

```python
x_new_quad = new_features(x, degree=2)
x_new_cubic = new_features(x, degree=3)

#reordering output features
temp = np.copy(x_new_quad[:, -1])
x_new_quad[:, -1] = x_new_quad[:, -2]
x_new_quad[:, -2] = temp

temp = np.copy(x_new_cubic[:, -1])
x_new_cubic[:, -1] = x_new_cubic[:, -2]
x_new_cubic[:, -2] = x_new_cubic[:, -3]
x_new_cubic[:, -3] = temp

```

Now use the gradient ascent optimization algorithm to learn theta by maximizing the log-likelihood, both for the case of x_new_quad and x_new_cubic.

```python
# Initialize theta0, in case of quadratic features
theta0_quad = np.zeros(x_new_quad.shape[1])

theta_final_quad, log_l_history_quad = gradient_ascent(theta0_quad,x_new_quad,y,lr=0.5,num_steps=n_iter)

# Initialize theta0, in case of quadratic and cubic features
theta0_cubic = np.zeros(x_new_cubic.shape[1])

# Run Newton's method, in case of quadratic and cubic features
theta_final_cubic, log_l_history_cubic = gradient_ascent(theta0_cubic,x_new_cubic,y,lr=0.5,num_steps=n_iter)

# check and compare with previous results
print(theta_final_quad)
print(theta_final_cubic)
```

    [ 0.07548038  3.32398303  0.27106753 -0.51873505 -0.34088582 -0.04846172]
    [ 0.82395453  2.41957969  1.66049077 -1.13927962 -0.20657699 -0.96074035
      0.34384121 -0.64078458  0.95132383  1.40927121]

```python
# Plot the log likelihood values in the optimization iterations, in one of the two cases.
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history_quad)),log_l_history_quad,'b.')
```

#### Exercise 2.b **(3 Points)** Plot the computed non-linear boundary and discuss the questions

First, define a boundary_function to compute the boundary equation for the input feature vectors $x_1$ and $x_2$, according to estimated parameters theta, both in the case of quadratic (theta_final_quad) and of quadratic and cubic features (theta_final_cubic). Refer for the equation to the introductory part of Question 2.

################# Do not write above this line #################

Your code here

```python
def boundary_function(x1_vec, x2_vec, theta_final):

    x1_vec, x2_vec = np.meshgrid(x1_vec,x2_vec)

    if len(theta_final) == 6:
        # boundary function value for features up to quadratic
        c_0, c_1, c_2, c_3, c_4, c_5 = theta_final
        # f =
    elif len(theta_final) == 10:
        # boundary function value for features up to cubic
        c_0, c_1, c_2, c_3, c_4, c_5, c_6, c_7, c_8, c_9 = theta_final
        # f =
    else:
        raise("Number of Parameters is not correct")

    return x1_vec, x2_vec, f
```

################# Do not write below this line #################

Now plot the decision boundaries corresponding to the theta_final_quad and theta_final_cubic solutions.

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_cubic)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

#### Confusion Matrix

Here you can see the confusion matrices related to the three models you've implemented.

```python
from sklearn.metrics import confusion_matrix
```

```python
## logistic regression with linear buondary

z = np.dot(x,theta_final)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[218,  35],
           [ 22, 225]], dtype=int64)

```python
## logistic regression with non linear buondary - quadratic

z = np.dot(x_new_quad,theta_final_quad)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[220,  33],
           [ 15, 232]], dtype=int64)

```python
## logistic regression with non linear buondary - cubic

z = np.dot(x_new_cubic,theta_final_cubic)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[225,  28],
           [ 11, 236]], dtype=int64)

################# Do not write above this line #################

Write now your considerations. Discuss in particular:
- Look back at the plots you have generated. What can you say about the differences between the linear, quadratic, and cubic decision boundaries? Can you say if the model is improving in performances, increasing the degree of the polynomial? Do you think you can incur in underfitting increasing more and more the degree?
- Let's now delve into some quantitative analysis. The three tables you have generated represent the confusion matrix for the model you have implemented in the first two questions. What can you say about actual performances? Does the increase of the degree have a high effect on the results?

################# Do not write below this line #################

## Question 3: Multinomial Classification (Softmax Regression) **(13 Points)**

### Code and Theory **(10 Points)**
### Report **(3 Points)**

#### Exercise 3.a **(4 Points)**

In the multinomial classification we generally have $K>2$ classes. So the label for the $i$-th sample $X_i$ is $y_i\in\{1,...,K\}$, where $i=1,...,N$. The output class for each sample is estimated by returning a score $s_i$ for each of the K classes. This results in a vector of scores of dimension K.
In this exercise we'll use the *Softmax Regression* model, which is the natural extension of *Logistic Regression* for the case of more than 2 classes. The score array is given by the linear model:

\begin{equation}
s_i =  X_i \theta
\end{equation}

Scores may be interpreted probabilistically, upon application of the function *softmax*. The position in the vector with the highest probability will be predicted as the output class. The probability of the class k for the $i$-th data sample is:

\begin{equation}
p_{ik} = \frac{\exp(X_i \theta_k)}{\sum_{j=1}^K(X_i \theta_j))}
\end{equation}

We will adopt the *Cross Entropy* loss and optimize the model via *Gradient Descent*.
In the first of this exercise we have to:
-    Write the equations of the Cross Entropy loss for the Softmax regression model;
-    Compute the equation for the gradient of the Cross Entropy loss for the model, in order to use it in the gradient descent algorithm.

#### A bit of notation

*  N: is the number of samples
*  K: is the number of classes
*  X: is the input dataset and it has shape (N, H) where H is the number of features
*  y: is the output array with the labels; it has shape (N, 1)
*  $\theta$: is the parameter matrix of the model; it has shape (H, K)

################# Do not write above this line #################

Your equations here.

\begin{equation}
L(\theta) = ...
\end{equation}

\begin{equation}
\nabla_{\theta_k} L(\theta) = ...
\end{equation}

################# Do not write below this line #################

#### Exercise 3.b **(4 Points)**

Now we will implement the code for the equations. Let's implement the functions:
-  softmax
-  CELoss
-  CELoss gradient
-  gradient descent

We generate a toy dataset with *sklearn* library. Do not change anything outside the parts provided of your own code (else the provided checkpoint will not work).

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=300, n_features=7, n_informative=7, n_redundant=0, n_classes=3, random_state=1)
X.shape, y.shape
```

    ((300, 7), (300,))

As a hint for the implementations of your functions: consider the labels $y$ as one-hot vector. This will allow matrix operations (element-wise multiplication and summation).

```python
import scipy
import numpy as np

def class2OneHot(vec):
    out_sparse = scipy.sparse.csr_matrix((np.ones(vec.shape[0]), (vec, np.array(range(vec.shape[0])))))
    out_onehot = np.array(out_sparse.todense()).T
    return out_onehot

y_onehot = class2OneHot(y)
```

Let's visualize the generated dataset. We use as visualizzation method the *Principal Component Analysis* (PCA). PCA summarize the high-dimensional feature vectors of each sample into 2  features, which we can illustrate with a 2D plot. Look at the following plot, the 3 generated classes do not seem separable.

```python
from sklearn.decomposition import PCA
import pandas as pd

pca = PCA(n_components=2)
principalComponents = pca.fit_transform(X)
principalDf = pd.DataFrame(data = principalComponents, columns = ['pc1', 'pc2'])
finalDf = pd.concat([principalDf, pd.DataFrame(y, columns = ['target'])], axis = 1)
```

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x='pc1', y='pc2', hue='target', data=finalDf);
```

################# Do not write above this line #################

```python
def softmax(theta, X):
    '''
    Function to compute associated probability for each sample and each class.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)

    Output:
    softmax: it's the matrix containing probability for each sample and each class. The shape is (N, K)
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return softmax

def CELoss(theta, X, y_onehot):
    '''
    Function to compute softmax regression model and Cross Entropy loss.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)

    Output:
    loss: The scalar that is the mean error for each sample.
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return loss

def CELoss_jacobian(theta, X, y_onehot):
    '''
    Function to compute gradient of the cross entropy loss with respect the parameters.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)

    Output:
    jacobian: A matrix with the partial derivatives of the loss. The shape is (H, K)
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return jacobian

def gradient_descent(theta, X, y_onehot, alpha=0.01, iterations=100):
    '''
    Function to compute gradient of the cross entropy loss with respect the parameters.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)
    alpha: it's the learning rate, so it determines the speed of each step of the GD algorithm
    iterations: it's the total number of step the algorithm performs

    Output:
    theta: it's the updated matrix of the parameters after all the iterations of the optimization algorithm. The shape is (H, K)
    loss_history: it's an array with the computed loss after each iteration
    '''

    # We initialize an empty array to be filled with loss value after each iteration
    loss_history = np.zeros(iterations)

    # With a for loop we compute the steps of GD algo
    for it in range(iterations):

        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

    return theta, loss_history
```

################# Do not write below this line #################

```python
# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))

print("Initial Loss with initialized theta is:", CELoss(theta0, X, y_onehot))

# Run Gradient Descent method
n_iter = 1000
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha=0.01, iterations=n_iter)
```

```python
theta_final
```

```python
loss = CELoss(theta_final, X, y_onehot)
loss
```

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

#### Exercise 3.c **(2 Points)**

Let's now evaluate the goodness of the learnt based on accuracy:

\begin{equation}
Accuracy = \frac{Number\ of\ correct\ predictions}{Total\ number\ of\ predictions}
\end{equation}

Implement the compute_accuracy function. You may compare the accuracy achieved with learnt model Vs. a random model (random $\Theta$) or one based on $\Theta$'s filled with zeros.

################# Do not write above this line #################

```python
def compute_accuracy(theta, X, y):
    '''
    Function to compute accuracy metrics of the softmax regression model.

    Input:
    theta: it's the final parameter matrix. The one we learned after all the iterations of the GD algorithm. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y: it's the label array. The shape is (N, 1)

    Output:
    accuracy: Score of the accuracy.
    '''

    return accuracy
```

################# Do not write below this line #################

```python
compute_accuracy(theta_final, X, y)
```

```python
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))
compute_accuracy(theta0, X, y)
```

```python
compute_accuracy(np.zeros((X.shape[1], len(np.unique(y)))), X, y)
```

### Report **(3 Points)**

Experiment with different values for the learning rate $\alpha$ and the number of iterations. Look how the loss plot changes the convergence rate and the resulting accuracy metric. Report also execution time of each run. For this last step you could you %%time at the beginning of the cell to display time needed for the algorithm.

```python
%%time

# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))

print("Initial Loss with initialized theta is:", CELoss(theta0, X, y_onehot))

# Run Gradient Descent method
n_iter = 100
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha=0.001, iterations=n_iter)
```

**Write your Report here**

| LR | Iter | Accuracy | Time |
|---|---|---|---|
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
| ... |  |  |  |

## Question 4: Multinomial Naive Bayes **(6 Points)**

### Code and Theory

The Naive Bayes classifier is a probabilistic machine learning model often used for classification tasks, e.g. document classification problems.

In the multinomial Naive Bayes classification you generally have $K>2$ classes, and the features are assumed to be generated from a multinomial distribution.

##### __*Example Data*__
General models consider input data as values. In the case of MultinomialNB, being used mainly in the field of document classification, these data consider how many features $X_i$ are present in the sample. Basically, it is a count of features within each document.

Taking into account $D=3$ documents and a vocabulary consisting of $N=4$ words, the data are considered as follows.

|  | $w_1$ | $w_2$ | $w_3$ | $w_4$ |
|---|---|---|---|---|
| $d_1$  | 3 | 0 | 1 | 1 |
| $d_2$ | 2 | 1 | 3 |0|
| $d_3$ | 2 | 2 | 0 |2|

By randomly generating the class to which each document belongs we have $y=[1,0,1]$

##### __*A bit of notation*__
- $Y =\{y_1, y_2, ... , y_{|Y|}\}$: set of classes

- $V =\{w_1, w_2, ... , w_{|V|}\}$: set of vocabulary

-  $D =\{d_1, d_2, ... , d_{|D|}\}$: set of documents

-  $N_{yi}$:  count of a specific word $w_i$ in each unique class, e.g. for $y=1$ you select $D_1$ and $D_3$, then for third column  you have $N_{y,3}=1$
-  $N_y$: total count of features for a specific class, e.g. for $y=1$ you sum all rows values which the correspondent label is 1, so $N_y=11$

-  $n$: total number of features (words in vocabulary)
-  $\alpha$: smoothing parameters

##### __*Task*__
Find the class $y$ to which the document is most likely to belong given the words $w$.
Use the Bayes formula and the posterior probability for it.

Bayes Formula:
\begin{equation}
P(A|B) = \frac{P(A)*P(B|A)}{P(B)}
\end{equation}

Where:
- P(A): Prior probability of A
- P(B): Prior probability of B
- P(B|A): Likelihood, multiplying posterior probability, that is multinomial Naive Bayes is:
\begin{equation}
P(B|A) = \left(\frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features}\right)^{X_{doc,i}}
\end{equation}

**Reminder: do not change any part of this notebook outside the assigned work spaces**

#### Generate random dataset

```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=300, n_features=7, n_informative=7, n_redundant=0, n_classes=3, random_state=1)
X = np.floor(X)-np.min(np.floor(X))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train.shape, y_train.shape
```

    ((240, 7), (240,))

#### Step0: $N_y$ and $N_{yi}$

```python
def feature_count(X, y, classes, n_classes, n_features):
        '''
        Function to compute the count of a specific word in each unique class and the total count.

        Input:
        X: it's the input data matrix.
        y: label array
        classes: unique values of y
        n_classes: number of classes
        n_features: it's the number of word in Vocabulary.

        Output:
        N_yi:   count of a specific word $w_i$ in each unique class
        N_y: total count of features for a specific class
        '''
        N_yi = np.zeros((n_classes, n_features)) # feature count
        N_y = np.zeros((n_classes)) # total count

        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        return N_yi, N_y
```

```python
n_samples_train, n_features = X_train.shape
classes = np.unique(y_train)
n_classes = 3
alpha = 0.1

N_yi, N_y = feature_count(X_train, y_train, classes, n_classes, n_features)
```

#### Step1: Prior Probability
The probability of a document being in a specific category from the given set of documents.

######################################

Your equations here:
\begin{equation}
P(y_j) = ...
\end{equation}

######################################

```python
def prior_(X, y, n_classes, n_samples):
        """
        Calculates prior for each unique class in y.

        Input:
        X: it's the input data matrix.
        y: label array
        n_classes: number of classes
        n_samples: number of documents

        Output:
        P: prior probability for each class. Shape: (, n_classes)
        """
        classes = np.unique(y)
        P = np.zeros(n_classes)

        # Implement Prior Probability P(A)
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        return P
```

```python
prior_prob = prior_(X_train, y_train, n_classes, n_samples_train)
print(prior_prob)
```

    [0.3        0.34583333 0.35416667]

#### Step2
Posterior Probability: The conditional probability of a word occurring in a document given that the document belongs to a particular category.

\begin{equation}
P(w_i|y_j) = \left(\frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features}\right)^{X_{doc,i}}
\end{equation}

Likelihood for a single document:
######################################

Your equations here:
\begin{equation}
P(w|y_j) = ...
\end{equation}

######################################

```python
def posterior_(x_i, i, h, N_y, N_yi, n_features, alpha):
        """
        Calculates posterior probability. aka P(w_i|y_j) using equation in the notebook.

        Input:
        x_i: feature x_i
        i: feature index.
        h: a class in y
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        posterior: P(xi | y). Float.
        """

        # Implement Posterior Probability
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################
        return posterior

def likelihood_(x, h, N_y, N_yi, n_features, alpha):
        """
        Calculates Likelihood P(w|j_i).

        Input:
        x: a row of test data. Shape(n_features,)
        h: a class in y
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        likelihood: Float.
        """

        tmp = []
        for i in range(x.shape[0]):
            tmp.append(posterior_(x[i], i, h, N_y, N_yi, n_features, alpha))

        # Implement Likelihood
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        return likelihood
```

```python
# Example of likelihood for first document
likelihood_(X_test[0], 0, N_y, N_yi, n_features, alpha)
```

    2.7754694679413126e-53

#### Step3
Joint Likelihood that, given the words, the documents belongs to specific class
######################################

Your equations here:
\begin{equation}
P(y_i|w) = ...
\end{equation}

######################################

Finally, from the probability that the document is in that class given the words, take the argument correspond to max value.

\begin{equation}
y(D) = argmax_{y \in Y} \frac{P(y|w)}{\sum_{j}P(y_j|w)}
\end{equation}

```python
def joint_likelihood(X, prior_prob, classes, n_classes, N_y, N_yi, n_features, alpha):
        """
        Calculates the joint probability P(y_i|w) for each class and makes it probability.
        Then take the argmax.

        Input:
        X: test data
        prior_prob:
        classes:
        n_classes:
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        predicted_class: Predicted class of the documents. Int. Shape: (,#documents)
        """
        samples, features = X.shape
        predict_proba = np.zeros((samples,n_classes))

        # Calculate Joint Likelihood of each row for each class, then normalize in order to make them probabilities
        # Finally take the argmax to have the predicted class for each document
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        return predicted_class
```

```python
yhat = joint_likelihood(X_test, prior_prob, classes, n_classes, N_y, N_yi, n_features, alpha)
```

#### Step4: Calculate the Accuracy Score

```python
print('Accuracy: ', np.round(accuracy_score(yhat, y_test),3))
```

    Accuracy:  0.717

**Sanity Check**

Here we use a function from the sklearn library, one of the most widely used in machine learning. MultinomialNB() implements the required algorithm, so the result of your implementation should be equal to the output of the following function.

```python
from sklearn import naive_bayes
clf = naive_bayes.MultinomialNB(alpha=0.1)
clf.fit(X_train,y_train)
sk_y = clf.predict(X_test)
print('Accuracy: ', np.round(accuracy_score(sk_y, y_test),3))
```

    Accuracy:  0.717