# Monte Carlo methods - A Sandbox/Demo

Monte Carlo broadly refers to random sampling methods. Here we will focus mainly on Monte Carlo sampling in the context of molecular simulations, but it's worth briefly considering an example of Monte Carlo integration. Particularly, we can use Monte Carlo integration to calculate $\pi$ with just a few lines of code.

## Preparation

To use this notebook you will either need to run it locally using Jupyter (with a fortran library you pre-compile) or on Google Colab. This notebook briefly explains either route, though if you are going the local route you may need to compile the library before installing some of the other course materials, or do so in a clean/new conda environment. Some users have reported that conda installing `openforcefield`, `openmm` and the openeye toolkits after installing `gfortran` results in the loss of the ability to compile fortran libraries for use in Python.

### Preparation for Google Colab (NOT FOR LOCAL USE)

[](https://colab.research.google.com/github/MobleyLab/drug-computing/blob/master/uci-pharmsci/lectures/MC/MC_Sandbox.ipynb)

For Google Colab, pip installation of software is faster, but we've only been able to get `gfortran` working via a conda installation, so we'll need to go that route. Begin by unsetting the PYTHONPATH to prevent issues with miniconda, then installing miniconda (which will take perhaps 20 seconds to a couple of minutes):

```python
%env PYTHONPATH=
! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.10.3-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.10.3-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.10.3-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.7/site-packages/')
```

Once that is done, install `gfortran`, which will take roughly a similar amount of time:

```python
!conda install -c conda-forge libgfortran --yes
```

Next, mount your Google Drive and ensure you have the mc_sandbox.f90 file available **at a path you define below**:

```python
from google.colab import drive
drive.mount('/content/drive',force_remount = True)

#EDIT THIS TO DEFINE WHERE YOU PUT THE FILE:
md_library_path = '/content/drive/MyDrive/drug-computing/uci-pharmsci/lectures/MC/'

# Then run:
%cd $md_library_path
```

Then compile the requisite library:

```python
!f2py3 -c -m mc_sandbox mc_sandbox.f90
```

### Preparation for local use

For local use, you need to compile the fortran module `mc_sandbox` for use as a Python library. This accelerates the numerical calculations and a similar framework will be used in the MD and MC assignments. Use `f2py3 -c -m mc_sandbox mc_sandbox.f90` to compile. (As noted above you may need to do this in advance of installing certain modules, such as the beginning of the course, or by making a clean conda environment.)

## Calculating $\pi$ via MC integration

So, let's calculate $\pi$ by doing MC integration. Particularly, let's consider a drawing random numbers between -1 and 1, and then imagine a circle centered at (0, 0). Consider the area of that circle (of radius 1) to that of the full square spanned by our random numbers (2 units wide). Particularly, if $R$ is the radius of the circle, then the ratio of the areas is:
\begin{equation}
\frac{A_{sq}}{A_{cir}} = \frac{(2R)^2}{\pi R^2} = \frac{4}{\pi}
\end{equation}

so we find
\begin{equation}
\pi = \frac{4 A_{cir}}{A_{sq}}
\end{equation}

So, if we randomly place points in an interval -1 to 1, and then check to see how many fall within a square versus within a circle, we can use the ratio of counts (related to the ratio of the areas) to determine $\pi$.

```python
#Import modules we need
import numpy
import numpy.random

#Number of data points to sample
Ntrials = 1000

#Randomly generate array of XY positions - spanning -1 to 1
XY = 2.*numpy.random.rand( Ntrials, 2)-1.

#Compute distance from each point to center of the circle
distances = numpy.sqrt( numpy.sum( XY*XY, axis=1))

#Find indices of data points which are within the unit circle
#Note that you could code this in a more straightforward - but slower - way by setting up a
#'for' loop over data points and checking to see where the distance is less than 1.
indices_inside = numpy.where( distances < 1)

#Find how many points are here
num_inside = len( indices_inside[0] )

#Calculate estimate of pi
pi_estimate = 4.*num_inside/float(Ntrials)
print(pi_estimate)
```

    3.152

### Let's take a look at what we've done here, graphically.
We can plot the points inside and outside in two different colors to see.

```python
%pylab inline

#After the code above, indices_inside has the numbers of points which are inside our unit circle.
#We would also like indices of points which are outside the unit circle
indices_outside = numpy.where( distances > 1)

#Make our plot of points inside and outside using circles for the data points
#blue for inside, red for outside
plot(XY[indices_inside[0],0], XY[indices_inside[0],1], 'bo')
plot(XY[indices_outside[0],0], XY[indices_outside[0],1], 'ro')

#Adjust plot settings to look nicer
axis('equal') #Set to have equal axes
F = pylab.gcf() #Get handle of current figure
F.set_size_inches(5,5) #Set size to be square (default view is rectangular)

#Bonus: Drawing a unit circle on the graph is up to you
```

# Let's revisit the LJ particles we saw in the MD sandbox

In our last sandbox, we looked at molecular dynamics on a pair of Lennard-Jones particles. Now let's revisit that, but within the Metropolis Monte Carlo framework. Optionally, you could make things a little more interesting here by considering an extra particle. But for now let's just start with the same two particles as last time to make visualization easy..

Here, we'll apply the Metropolis MC framework as discussed in lecture, where every step we:
* Randomly pick a particle
* Change each of x, y, and z a small amount between $-\Delta r_{max}$ and $\Delta r_{max}$
* Compute the energy change $\Delta U$
* Apply the Metropolis criterion to decide whether to accept the move or reject it
  * If $\Delta U < 0$, accept the move
  * If $\Delta U > 0$, accept with the probability $P_{acc} = e^{-\Delta U/T}$
* If accepted, keep the new configuration
* Regardless of whether we accept it or not, update any running averages with the current state and energy

## Let's run some MC

(As noted above, you will need to have compiled the `mc_sandbox` Fortran library before doing this, using either the appropriate technique for local work or for on Colab.)

### First, we set up our system:

```python
import mc_sandbox
import numpy as np
import numpy.random

#Let's define the variables we'll need
Cut = 2.5
L = 3.0 #Let's put these in a small box so they don't lose each other
max_displacement = 0.1 #Maximum move size
T = 1. #You should play with this and see how the results change. Don't make it an integer - use a floating point value (i.e. 1., not 1)

#Choose N for number of particles; you could adjust this later.
N = 2

#Allocate position array - initially just zeros
Pos = np.zeros((N,3), float)

#Let's place the first two particles just as we did in MD Sandbox:
Pos[0,:] = np.array([0,0,0])
#We'll place the second one fairly nearby - at this point using the same starting location
#as in the MD sandbox
Pos[1,:] = np.array([1.5,0,0])

#If you have any other particles, let's just place them randomly
for i in range(2,N):
    Pos[i,:] = L*np.random.random( 3 )
```

### Now let's run a step of MC!

```python
#Set maximum number of steps to run
max_steps = 10000

#Set up storage for position vs step
Pos_t = np.zeros(( N,3,max_steps), float)
#Store initial positions
Pos_t[:,:,0] = Pos

#Evaluate initial energy
U = mc_sandbox.calcenergy(Pos, L, Cut)

#Pick a random particle
num = np.random.randint(N) #Random integer from 0 up to but not including N

#Store old position in case we need to revert
#Note that it's necessary here to make a copy, otherwise both still point to the same
#coordinates (try OldPos = Pos to see).
OldPos = Pos.copy()

#Pick a move - adjusting to make it between -DeltaX and +DeltaX
move = max_displacement * (np.random.random( 3)*2.-1.)

#Update position
Pos[num, :] += move

#Evaluate new energy
Unew = mc_sandbox.calcenergy( Pos, L, Cut)
DeltaU = Unew - U
print("U, Unew, DeltaU: ", U, Unew, DeltaU) #Just for debugging purposes so we can see what's happening.

#Print acceptance probability
Pacc = np.exp(-DeltaU/T)
print("Acceptance probability Pacc=", Pacc) #Just for debugging purposes so we can see what's happening.

#We can handle the uphill and downhill cases with a single 'if' statement
if np.random.rand() < Pacc:
    print("Accepted")  #Just for debugging purposes so we can see what's happening.
    U = Unew
else: #Revert
    Pos = OldPos
    print("Rejected") #Just for debugging purposes so we can see what's happening.

#Remember, at the end, if we are tracking energy, we update running averages/tracking data
#with the current position and energy
Pos_t[:, :, 1] = Pos
```

    U, Unew, DeltaU:  -0.3366534854145746 -0.42428826461302754 -0.08763477919845292
    Acceptance probability Pacc= 1.0915893780702166
    Accepted

## Now let's again define that get_r function so we can look at the separation between our particles over time
Last time, I was lazy and didn't handle the minimum image convention (in part because I knew we weren't giving the particles enough energy initially that they wouldn't fly apart). Here, because T (effectively, the kinetic energy) is an adjustable parameter we might end up with them flying apart, so we need to use the minimum image convention to properly measure the distance between particles. In other words, if a particle crosses the box edge and then finds the other particle and interacts with it, we want our distance measurement to notice that they are interacting rather than reporting that they are very distant. (The Fortran code we have is using this convention for its energy calculations).

So, we define a new get_r function which handles this:

```python
def get_r(Pos, L):
    """Calculate r, the distance between particles, for a position array containing just two particles. Return it.
    Unlike in MD sandbox, here we also implement the minimum image convention"""

    #Get displacement
    disp = Pos[1,:] - Pos[0,:]
    #Apply minimum image convention
    disp = disp - L*np.round(disp/L)
    #Calculate distance
    d = np.sqrt( np.dot( disp, disp))
    return d
```

## Now, write your own code - adapting the code from above for a single step - to run an MC simulation of your pair of LJ particles

Because a little more code is required than for the MD assignment, I provide some comments guiding you through the steps you'll need to do. And I also provide code to generate a plot at the end.

Be sure to track the acceptance probability over all suggested moves. (This could be used to adjust the size of the moves you suggest).

```python
# Put your code here

```

```python
##GET READY TO PLOT
#Find x axis (MC steps rather than time, as it was in the MD sandbox)
t = np.arange(0,max_steps)
#Find y axis (r values)
r_vs_t = []
for i in range(max_steps):
    r=get_r(Pos_t[:,:,i], L)
    r_vs_t.append(r)

r_vs_t = np.array(r_vs_t)

#Plot
figure()
plot(t, r_vs_t)
```

## Other things to be sure to try
* See what happens if you adjust the temperature. Particularly, check what happens in the limit of T becoming very small (approaching zero). Are uphill moves ever accepted? What does an MC search end up doing?
* Try making the box size reasonably big and see what happens at a moderate temperature
* Adjust the move size (`max_displacement`) to make the acceptance probability 30-50%. How big can you make it?

## Other things to try if you have extra time
* Check the probability distribution of separations and see how it varies with temperature.

## Simulation best practices

Scott Shell has some very useful [simulation best practices](https://sites.engineering.ucsb.edu/~shell/che210d/Simulation_best_practices.pdf) tips which can help with thinking through how to code up and conduct effective simulations.


This notebook holds the design parameters and generates an audio chirp

```python
%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.gridspec import GridSpec
from matplotlib.ticker import ScalarFormatter
import math
```

This notebook assumes you have completed the notebook [Introduction of sine waves](TDS_Introduction-sine_waves.ipynb). This notebook follows the same pattern of time domain waveform generation: instantaneous frequency -> angle step -> total angle -> time domain waveform.

Our goal is to track features of different acoustic impedance in material using a low power time domain waveform. Time delay spectrometry (TDS) is one implementation of this goal. To understand TDS we need to understand the waveform which is used by TDS called a chirp. A chirp is a sinusoid that is constantly varying in frequency. The chirp is generated by integrating a varying angle step which is derived from an instantaneous frequency profile. We will generate a chirp in this notebook. An overview of this technique is given [here](https://www.youtube.com/watch?v=RQplkt0bw_c).

The angle of the chirp can be found by integrating the instantaneous frequency:

\begin{equation}
	f(t)=\frac{f_{end}-f_{start}}{T_c}t + f_{start}
\end{equation}

\begin{equation}
\Delta\phi(t) = 2\pi f(t)\Delta t
\end{equation}

\begin{equation}
\phi (t)=\int_{}^{} \Delta\phi(t) = \int_{}^{} 2\pi f(t) dt = \int_{}^{}\frac{f_{end}-f_{start}}{T_c}tdt + \int_{}^{}f_{start}dt
\end{equation}

\begin{equation}
 \phi (t)= \frac{f_{end}-f_{start}}{T_c}\int_{}^{}tdt + f_{start}\int_{}^{}dt
\end{equation}

\begin{equation}
 \phi (t)= \frac{f_{end}-f_{start}}{T_c}\frac{t^2}{2} + f_{start}t
\end{equation}

This gives the time series value of

\begin{equation}
x(t) = e^{j\phi (t)} = e^{j(\frac{f_{end}-f_{start}}{T_c}\frac{t^2}{2} + f_{start}t)}
\end{equation}

But the formula for angle requires squaring time which will cause numeric errors as the time increases. Another approach is to implement the formula for angle as a cummulative summation.

\begin{equation}
\phi_{sum} (N)=\sum_{k=1}^{N} \Delta\phi(k) = \sum_{k=1}^{N} 2\pi f(k) t_s = \sum_{k=1}^{N}(\frac{f_{end}-f_{start}}{T_c}k + f_{start})t_s
\end{equation}

This allow for the angle always stay between 0 and two pi by subtracting two phi whenever the angle exceeds the value. We will work with the cummlative sum of angle, but then compare it to the integral to determine how accurate the cummulative sum is.

```python
#max free 8 points per sample

#Tc is the max depth we are interested in
Tc_sec=1

speed_of_sound_in_air_m_per_sec=343

f_start_Hz=3e3
#talk about difference and similarity of sine wave example, answer why not 32 samples
f_stop_Hz=20e3

print(f"The wavelength ranges from {(speed_of_sound_in_air_m_per_sec/f_start_Hz):.3f}m to {speed_of_sound_in_air_m_per_sec/f_stop_Hz:.3f} m")

#We choose 8 samples per cycle at the maximum frequency to not require steep pulse shaping filter profiles on the output of the
#digital to analog converter
fs=44.1e3

samplesPerCycle=fs/f_stop_Hz

ts=1/fs

total_samples= math.ceil(fs*Tc_sec)
n = np.arange(0,total_samples, step=1, dtype=np.float64)
t_sec=n*ts

#This is the frequency of the chirp over time. We assume linear change in frequency
chirp_freq_slope_HzPerSec=(f_stop_Hz-f_start_Hz)/Tc_sec

#Compute the instantaneous frequency which is a linear function
chirp_instantaneous_freq_Hz=chirp_freq_slope_HzPerSec*t_sec+f_start_Hz
chirp_instantaneous_angular_freq_radPerSec=2*np.pi*chirp_instantaneous_freq_Hz

#Since frequency is a change in phase the we can plot it as a phase step
chirp_phase_step_rad=chirp_instantaneous_angular_freq_radPerSec*ts

#The phase step can be summed (or integrated) to produce the total phase which is the phase value
#for each point in time for the chirp function
chirp_phase_rad=np.cumsum(chirp_phase_step_rad)
#The time domain chirp function
chirp = np.exp(1j*chirp_phase_rad)
```

    The wavelength ranges from 0.114m to 0.017 m

```python
#We can see, unlike the complex exponential, the chirp's instantaneous frequency is linearly increasing.
#This corresponds with the linearly increasing phase step.
fig, ax = plt.subplots(2, 1, sharex=True,figsize = [8, 8])
lns1=ax[0].plot(t_sec,chirp_instantaneous_freq_Hz,linewidth=4, label='instantanous frequency');
ax[0].set_title('Comparing the instantaneous frequency and phase step')
ax[0].set_ylabel('instantaneous frequency (Hz)')

axt = ax[0].twinx()
lns2=axt.plot(t_sec,chirp_phase_step_rad,linewidth=2,color='black', linestyle=':', label='phase step');
axt.set_ylabel('phase step (rad)')

#ref: https://stackoverflow.com/questions/5484922/secondary-axis-with-twinx-how-to-add-to-legend
lns = lns1+lns2
labs = [l.get_label() for l in lns]
ax[0].legend(lns, labs, loc=0)

#We see that summing or integrating the linearly increasing phase step gives a quadratic function of total phase.
ax[1].plot(t_sec,chirp_phase_rad,linewidth=4,label='chirp');
ax[1].plot([t_sec[0], t_sec[-1]],[chirp_phase_rad[0], chirp_phase_rad[-1]],linewidth=1, linestyle=':',label='linear (x=y)');
ax[1].set_title('Cumulative quandratic phase function of chirp')
ax[1].set_xlabel('time (sec)')
ax[1].set_ylabel('total phase (rad)')
ax[1].legend();

```

    <IPython.core.display.Javascript object>

```python
#Write a wav file that is 32-bit floating-point [-1.0,+1.0] np.float32
from scipy.io.wavfile import write
write(f'../data/audio_chirp_{t_sec}sec_{f_start_Hz}Hz_{f_stop_Hz}Hz.wav', int(fs), np.real(chirp).astype(np.float32))
```


## RSA (Rivest–Shamir–Adleman) - Cryptosystem

### Algorithm

#### 1. Choose two distinct prime numbers p and q.

 For security purposes, the integers p and q should be chosen at random, and should be similar in magnitude but differ in length by a few digits to make factoring harder. Prime integers can be efficiently found using a primality test.

#### 2. Compute n = pq.

 n is used as the modulus for both the public and private keys. Its length, usually expressed in bits, is the key length.

#### 3. Compute Φ(n) = lcm(Φ(p), Φ(q)) = lcm(p − 1, q − 1)

 Φ is Carmichael's totient function. This value is kept private.

#### 4. Choose an integer e such that 1 < e < Φ(n) and gcd(e, Φ(n)) = 1; i.e., e and Φ(n) are coprime.

 Note that if e is prime and also not a divisor of Φ(n), so e and Φ(n) are coprime.

#### 5. Determine d as (e*d) %  Φ(n) = 1

 d is the modular multiplicative inverse of e (modulo Φ(n)).

#### To encrypt:

 c(m) = m^e % n

 The public key, composed by e and n, is used to encrypt the message.

#### To decrypt:

 m(c) = m^d % n

 The private key, composed by d and n, is used to decrypt the encrypted message.

#### References:

[Wikipedia - RSA (cryptosystem)](https://en.wikipedia.org/wiki/RSA_%28cryptosystem%29)

[Wikipedia - Carmichael function](https://en.wikipedia.org/wiki/Carmichael_function)

[Sympy.org](http://www.sympy.org/pt/index.html)

[Wikibooks - Extended Euclidean algorithm](https://en.wikibooks.org/wiki/Algorithm_Implementation/Mathematics/Extended_Euclidean_algorithm)

```python
from random import randint
import sympy
import numpy

# return the smallest prime in range [n,n+1000)
# return -1 if it doesn't exist
def next_prime(n):
    for i in range(n,n+1000):
        if (sympy.isprime(i)):
            return i
    return -1

# Extended Euclidean algorithm
def egcd(a, b):
    if a != 0:
        x, y, z = egcd(b % a, a)
        return (x, z - (b // a) * y, y)
    return (b, 0, 1)

# Modular Inverse
# return -1 if it doesn't exist
def mod_inv(a, b):
    x, y, _ = egcd(a, b)
    if x != 1:
        return -1
    else:
        return y % b

def encrypt(m,e,n):
    return pow(m,e,n)

def decrypt(c,d,n):
    return pow(c,d,n)

## 1. p and q

p = -1
while(p == -1):
    p = next_prime(randint(10**130,10**150))

q = -1
while(q == -1):
    q = next_prime(randint(10**100,10**120))
    if (q == p):
        q = -1

if(randint(0,9) % 2 == 0):
    aux = p
    p = q
    q = aux

print("p and q:")
print(hex(p))
print(hex(q))

## 2. n

n = p*q

print("\nn:")
print(hex(n))

## 3. phi

phi = sympy.lcm((p-1),(q-1))

print("\nphi(n):")
print(hex(phi))

## 4. e

e = -1
while(e == -1):
    e = next_prime(randint(2,phi))
    if e >= phi:
        e = -1
    elif e%phi == 0:
        e = -1

print("\ne:")
print(hex(e))

## 5. d

d = int(mod_inv(e, phi))

if d == -1:
    raise Exception('Modular inverse does not exist, but don`t worry, just run it again. =)')

print("\nd:")
print(hex(d))

## Encrypt:

c = encrypt(42,e,n)
print("\nmessage encrypted:")
print(hex(c))

## Decrypt:

m = decrypt(c,d,n)
print("\nmessage decrypted:")
print(m)

```

    p and q:
    0x4972224d37eeec694dbc80cdb3b88052a29faa76b113148ad06f89995c3a7fcdb97cbe523efadde3a0b889e7f8094b893b255c5c7a05d7c99819c9c3e9d29
    0x3e5810c28da948c519eb2b92dccc0f1afc654f60bc1ad1588514a5cf20b0ee057117cc0071499f887d56e9477be11a12f23b

    n:
    0x11e2e859715e5783e98e5c5d6d5d73817889d24062c2f6c3212a495b072cc1a30fbac63d61d0221094d0c31731a3d4ed56a36d57374822025b1f9626e90c087607d3da2127815f5f38216729e9fe211b8c6e37b702828131a57555c8d0b23f3d584e65118d537fc2f28b2173169e0fa73

    phi(n):
    0x8f1742cb8af2bc1f4c72e2eb6aeb9c0bc44e92031617b61909524ad839660d187dd631eb0e811084a68618b98d1ea76ab5191f28a7d7509b75b2435341f286c033a8753743f2e56b86eb476cfb98a8d5bd5393c4b8d5cf2be9a61829bf4f8114802916bcb7f3ef3d61688cd9d9c7b588

    e:
    0x5645c37219fad185353336fd4f4aa8a1d95ecfe4d083ef88e064082ddc892eee4db2127436c07be7edeed3fb96cf941531ac9ea5bb4b17bb5bb8854b42c67f6998038558b7f3033cf2c84cfd598a3218e3d1fa34901767143c26c744d80bc88fb691ebfe975279f1aba94fd56924ce9d

    d:
    0x3d0cf1214c4b09200b502b82b053d719e78bb40a1bbfbe8314a4faaa0fecd828273b559921ba436d6ae979c7d6aeedac2255a76ec54d61e0476091101b2f2c3de4a31e2f00061ad351bad807dfbe4a55f5e3e4ef94e3b65f89f3d03ba7f79fadfb17a7a9113462fa256b4c2201f32a6d

    message encrypted:
    0x1036f1372d8acabac7b61cc594e93393c9a30754d8270134da09a71cdbd1aa62591f132ef877a21e595996b90ab8fb3f2361afd44a5665f8ff51f86a1ea35f2f5144d9afeac125c48696bb3fd6dd034a3cffb423cc57d2a706cbbc6833344272be898d53d14bdc459b6b958128d96beaa

    message decrypted:
    42


### Probabilistic model
Naive Bayes models the conditional probability of classes $C$, given an instance represented by a feature vector $x=(x_1, \dots, x_n)$, as
\begin{align}
p(C_k \mid x) = \frac{p(x \mid C) p(C)}{p(x)}.
\end{align}
The most important assumption of the Naive Bayes model is that it assumes that all features are mutually independent conditional on the category $C$, e.g.,
\begin{align}
p(x_i \mid x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n, C) = p(x_i \mid C).
\end{align}

### Naive Bayes classifier
The Naive Bayes classifier is based on the MAP (maximum a posteriori) estimate of the conditional probability $p(C, x)$, i.e., given a feature vector $x$, we predict it being of the class
\begin{align}
\hat{y} = \text{argmax}_{k \in [K]} p(C) \prod_{i=1}^n p(x_i \mid C),
\end{align}
or equivalently (for computational reasons)
\begin{align}
\hat{y} = \text{argmax}_{k \in [K]} \left[ \log \left( p(C) \right) + \sum_{i=1}^n \log \left( p(x_i \mid C\right) \right].
\end{align}
For simplicity we assume here that $p(C=k) = c_k$ is constant.

#### Modeling  the conditional probabilities
One can choose any model for the conditional probabilities $p(x_i \mid C)$, e.g., Gaussian, Bernoulli, multinomial, etc. Note that the Naive Bayes classifier can easily handle mixtures of categorical and real-valued features.

### Maximum likelihood training
Given a dataset $\left\{\left(X^{(j)}, Y^{(j)}\right)\right\}_{j \in [m]}$, we can train the parameters $\{ \theta_p \}_{p \in [P]}$ of our model (hidden in $p(x_i \mid C)$) using maximum likelihood, i.e., setting derivatives of the likelihood function $p(\theta | x, C)$ with respect to the $\theta_p$'s to zero and solving for the $\theta_p$'s. Using the independence assumption, we can find closed-form maximum likelihood estimates, e.g., if $p(x_i \mid C=k)$ is Gaussian, then
\begin{align}
\mu_{ik} &= \sum_{Y_j = k \; \forall j \in [m]} \frac{X^{(j)}_i}{n_k}, \\
\sigma_{ik} &= \sum_{Y_j = k \; \forall j \in [m]} \frac{(X^{(j)}_i - \mu_{ik})^2}{n_k},
\end{align}
where $n_k$ is the amount of samples with class $k$ in the dataset.

### Sampling
Due to the independence assumption, sampling examples $\left(\hat{x}, \hat{C}\right) \sim p(x, C)$ is fairly easy. We first set $p(C=k) = n_k / m$ and sample $\hat{C} \sim p(C)$. Afterwards, we can sample the features $\hat{x}_i \sim p\left(x_i \mid C = \hat{C}\right)$. Sampling methods for easy probability distributions are implemented in NumPy.

```python
import numpy as np
```

```python
class NaiveBayes:
    def __init__(self, class_prior, features, n_classes):
        self.class_prior = class_prior
        self.features = features
        self.n_classes = n_classes

        self._init_weights()

    def _init_weights(self):
        self.weights = []

        for i, feature in enumerate(self.features):
            if feature == 'gaussian':
                self.weights.append((self._random_normal(), self._random_normal()))

            elif feature == 'bernoulli':
                self.weights.append((self._random_uniform()))

    def _random_normal(self, loc_in=0.0, scale_in=1.0, size_in=None):
        if size_in is None:
            size_in = self.n_classes

        return np.random.normal(loc=loc_in, scale=scale_in, size=size_in)

    def _random_uniform(self, low_in=0.0, high_in=1.0, size_in=None):
        if size_in is None:
            size_in = self.n_classes

        return np.random.uniform(low=low_in, high=high_in, size=size_in)

    def _random_bernoulli(p_in, n_in=1, size_in=1):
        return np.random.multionimal(n=n_in, pvals=p_in, size=size_in)

    def _conditional_log_probability(self, feature, class_index, feature_index):
        if self.features[feature_index] == 'gaussian':
            return - (1/2)*np.log(2*np.pi*self.weights[feature_index][1][class_index]**2) \
                   - (feature-self.weights[feature_index][0][class_index])**2 \
                   / (2*self.weights[feature_index][1][class_index]**2)

        elif self.features[feature_index] == 'bernoulli':
            if feature == 0:
                return np.log(1-self.weights[feature_index][class_index])
            else:
                return np.log(self.weights[feature_index][class_index])

    def log_likelihood(self, X, Y):
        log_likelihood = 0.0

        for x, y in zip(X, Y):
            sum_of_logs = 0.0
            for j, _ in enumerate(self.features):
                sum_of_logs += np.log(self.class_prior[y]) + self._conditional_log_probability(x[j], y, j)

            log_likelihood += sum_of_logs

        return log_likelihood

    def _gaussian_maximum_likelihood_fit(self, feature_number, X, Y):
        means = np.zeros((self.n_classes,))
        counts = np.zeros((self.n_classes,))

        for x, y in zip(X, Y):
            means[y] += x
            counts[y] += 1

        for i in range(self.n_classes):
             means[i] /= counts[i]

        variances = np.zeros((self.n_classes,))

        for x, y in zip(X, Y):
            variances[y] += (x - means[y])**2 / counts[y]

        for i in range(self.n_classes):
            self.weights[feature_number][0][i] = means[i]
            self.weights[feature_number][1][i] = np.sqrt(variances[i])

    def _bernoulli_maximum_likelihood_fit(self, feature_number, X, Y):
        means = np.zeros((self.n_classes,))
        counts = np.zeros((self.n_classes,))

        for x, y in zip(X, Y):
            means[y] += x
            counts[y] += 1

        for i in range(self.n_classes):
             means[i] /= counts[i]

        for i in range(self.n_classes):
            self.weights[feature_number][i] = means[i]

    def maximum_likelihood_fit(self, X, Y):
        for j, feature in enumerate(self.features):
            if feature == 'gaussian':
                self._gaussian_maximum_likelihood_fit(j, X[:, j], Y)
            elif feature == 'bernoulli':
                self._bernoulli_maximum_likelihood_fit(j, X[:, j], Y)

    def predictions(self, X):
        predictions = []

        for x in X:

            class_predictions = []
            for i in range(self.n_classes):

                class_predictions.append(self.log_likelihood([x], [i]))

            predictions.append(class_predictions)

        return predictions

    def sampling(self, n_samples):
        samples = np.empty((n_samples, len(self.features) + 1))
        for i in range(n_samples):
            features = []
            ### sample a class
            sampled_class = np.argwhere(np.random.multinomial(n=1, pvals=self.class_prior) == 1)
            samples[i, -1] = sampled_class

            for j, feature in enumerate(self.features):
                if feature == 'gaussian':
                    samples[i, j] = self._random_normal(loc_in = self.weights[j][0][sampled_class][0],
                                                        scale_in = self.weights[j][1][sampled_class][0],
                                                        size_in = 1)
                elif feature == 'bernoulli':
                    samples[i, j] = self._random_bernoulli(self.weights[j][samples_class][0])

        return samples
```

We will now fit a Naive Bayes classifier Iris flower data set and afterwards try to sample points
from our generative model.

```python
from sklearn.datasets import load_iris
from sklearn.utils import shuffle
from sklearn import preprocessing
from scipy.special import softmax
import matplotlib.pyplot as plt
%matplotlib inline

X, Y = load_iris(return_X_y=True)
X = preprocessing.scale(X)
X, Y = shuffle(X, Y)

X_train = X[:100, :]
Y_train = Y[:100]

X_test = X[100:, :]
Y_test = Y[100:]

### Maximum likelihood fitting
naive_bayes_iris = NaiveBayes([1/3.]*3, ['gaussian']*4, 3)

print('Log-likelihood before fitting (random weights):', naive_bayes_iris.log_likelihood(X_train, Y_train))

naive_bayes_iris.maximum_likelihood_fit(X_train, Y_train)

print('Log-likelihood after fitting:', naive_bayes_iris.log_likelihood(X_train, Y_train))
```

    Log-likelihood before fitting (random weights): -1584.5829832421746
    Log-likelihood after fitting: -627.6014130704741

```python
predictions = naive_bayes_iris.predictions(X_test)
print('Accuracy on test set:', np.sum(np.argmax(predictions, axis=1) == Y_test) / 50)
```

    Accuracy on test set: 0.96

```python
### Sampling points
samples = naive_bayes_iris.sampling(50)
```

In the next plots we compare the data distribution to the sampling distribution. We can clearly see that the features (conditioned on the class) of the data distribution are correlated, and therefore the naive Bayes model might not be perfect for this case.

```python
### Data distribution for feature 1 and 2
plt.scatter(X_test[:, 0], X_test[:, 1], c=Y_test)
```

```python
### Sampling distribution for feature 1 and 2
plt.scatter(samples[:, 0], samples[:, 1], c=samples[:, 4])
```

```python
### Data distribution for feature 1 and 3
plt.scatter(X_test[:, 0], X_test[:, 2], c=Y_test)
```

```python
### Sampling distribution for feature 1 and 3
plt.scatter(samples[:, 0], samples[:, 2], c=samples[:, 4])
```


# Logistic regression with COSMO

_The presented example is adapted from the [Machine Learning - course by Andrew Ng](https://www.coursera.org/learn/machine-learning)._

In this example we use logistic regression to estimate the parameters $\theta_i$ of a logistic model in a classification problem. We will first transform the logistic regression problem into an exponential cone optimisation problem. We will then solve the optimisation problem with COSMO and determine the model parameters and the decision boundary.

### Visualizing the data

Before we start, let's load and take a look at the example data from `examples/chip_data.txt`:

```julia
using PyCall, LinearAlgebra, SparseArrays, COSMO, JuMP
pygui(:qt5)
using PyPlot

# load example data
f = open("./chip_data.txt")
lines = readlines(f)
close(f)
n_data = length(lines)
x1 = zeros(n_data)
x2 = zeros(n_data)
y = zeros(Float64, n_data)
for (i, l) in enumerate(lines)
    s = split(l, ",")
    x1[i] = parse(Float64, s[1])
    x2[i] = parse(Float64, s[2])
    y[i] = parse(Float64, s[3])
end

# visualize data
PyPlot.figure(1)
p1 = plot(x1[findall(x -> x == 1., y)], x2[findall(x -> x == 1., y)], marker = "+", color = "b", linestyle= "None", label="Accepted")
p2 = plot(x1[findall(x -> x != 1., y)], x2[findall(x -> x != 1., y)], marker = ".", color = "r", linestyle= "None", label="Rejected")
axis("equal")
PyPlot.grid(true)
PyPlot.legend()
xlabel("x1 - Microchip Test Score 1")
ylabel("x2 - Microchip Test Score 2")
nothing
```

The plot shows two test scores of $n$ microchip samples from a fabrication plant and whether the chip passed the quality check. Based on this data we would like to build a logistic model that takes into account the test scores and helps us predict the likelyhood of a chip being accepted.

### Defining the logistic model

The logistic regression hypothesis is given by

\begin{equation}
h_\theta(x) = g(\theta^\top x)
\end{equation}

where $g$ is the sigmoid function:

\begin{equation}
g(\theta^\top x) = \frac{1}{1+\exp(-\theta^\top x)}.
\end{equation}

$x$ are the independent variables and $\theta$ are the model parameters. For our samples we set the dependent variable $y =1$ if the chip was accepted and $y = 0$ otherwise.

$h_\theta(x)$ can be interpreted as the probability of the outcome being true rather than false. We want to find the parameters $\theta$ such that the following likelyhood function is maximized:

\begin{equation}
\text{maximize} \quad J(\theta) = h_\theta(x_i)^{y_i} (1-h_\theta(x_i))^{(1-y_i)} + \mu \|\theta \|_2,
\end{equation}

where we added a regularization term with parameter $\mu$ to prevent overfitting.

### Feature mapping
As our dataset only has two independent variables (the test scores) our model $y = \theta_0 + \theta_1 x_1 + \theta_2 x_2$ will have the form of a straight line. Looking at the plot one can see that a line will not perform well in separating the samples. Therefore, we will create more features based on each data point by mapping the original features ($x_1$, $x_2$) into all polynomial terms of $x_1$ and $x_2$ up to the 6th power:

\begin{equation}
\text{map_feature}(x_1,x_2) = [1, x_1, x_2, x_1^2, x_1x_2, x_2^2, x_1^3, \dots, x_1x_2^5, x_2^6 ]
\end{equation}

This will create 28 features for each sample.

```julia
function map_feature(x1, x2)
  deg = 6
  x_new = ones(length(x1))
  for i = 1:deg, j = 0:i
      x_new = hcat(x_new, x1.^(i-j) .* x2.^j)
  end
  return x_new
end

X = map_feature(x1, x2);
size(X)
```

    (118, 28)

### Transformation into a conic optimisation problem
We can rewrite above likelyhood maximisation problem as a conic optimisation problem with exponential-cone-, second-order-cone-, equality-, and inequality constraints:

\begin{equation}
\begin{array}{ll}
\text{minimize}  &\sum_i^n \epsilon_i + \mu v\\
\text{subject to}  & \|\theta \|_2 \leq v\\
&  \log(1 + \exp(-\theta^\top x_i)) \leq \epsilon_i  \quad \text{if } y_i = 1, \\
& \log(1 + \exp(\theta^\top x_i)) \leq \epsilon_i  \quad\text{   otherwise.}
\end{array}
\end{equation}

Implementing the constraint $\log(1 + \exp(z)) \leq \epsilon $ for each of the $n$ samples requires two exponential cone constraints, one inequality constraint and two equality constraints. To see this, take the exponential on both sides and then divide by $\exp(\epsilon)$ to get:

\begin{equation}
\exp(-\epsilon) + \exp(z - \epsilon) \leq 1.
\end{equation}

This constraint is equivalent to:

\begin{equation}
\begin{array}{ll}
(z - \epsilon, s_1, t_1) &\in K_{\text{exp}}, \\
(-\epsilon, s_2, t_2) &\in K_{\text{exp}}, \\
t_1 + t_2 &\leq 1,\\
s_1 = s_2 &= 1,
\end{array}
\end{equation}

where we defined the exponential cone as:

\begin{equation}
K_{\text{exp}} = \{(r, s, t) \mid s >0, s \exp(r/s) \leq t \} \cup \{ r \leq 0, s = 0, t \geq 0 \}.
\end{equation}

Based on this transformation our optimisation problem will have $5n + n_\theta + 1$ variables, 1 SOCP constraint, $2n$ exponential cone constraints, $n$ inequality constraints and $2n$ equality constraints.
Let's model the problem with JuMP and COSMO:

```julia
n_theta = size(X, 2)
n = n_data
μ  = 1.

m = Model(with_optimizer(COSMO.Optimizer))
@variable(m, v)
@variable(m, θ[1:n_theta])
@variable(m, e[1:n])
@variable(m, t1[1:n])
@variable(m, t2[1:n])
@variable(m, s1[1:n])
@variable(m, s2[1:n])

@objective(m, Min, μ * v + sum(e))
@constraint(m, [v; θ] in MOI.SecondOrderCone(n_theta + 1))

# create the constraints for each sample points
for i = 1:n
  yi = y[i]
  x = X[i, :]
  yi == 1. ? (a = -1) : (a = 1)
  @constraint(m, [a * dot(θ, x) - e[i]; s1[i]; t1[i] ] in MOI.ExponentialCone())
  @constraint(m, [-e[i]; s2[i]; t2[i]] in MOI.ExponentialCone())
  @constraint(m, t1[i] + t2[i] <= 1)
  @constraint(m, s1[i] == 1)
  @constraint(m, s2[i] == 1)
end
JuMP.optimize!(m)
theta = value.(θ)

```

    ------------------------------------------------------------------
                 COSMO - A Quadratic Objective Conic Solver
                             Michael Garstka
                    University of Oxford, 2017 - 2019
    ------------------------------------------------------------------

    Problem:  x ∈ R^{619},
              constraints: A ∈ R^{1091x619} (4513 nnz), b ∈ R^{1091},
              matrix size to factor: 1710x1710 (2924100 elem, 10736 nnz)
    Sets:     ZeroSet{Float64} of dim: 236
              Nonnegatives{Float64} of dim: 118
              SecondOrderCone{Float64} of dim: 29
              ExponentialCone{Float64} of dim: 3
              ExponentialCone{Float64} of dim: 3
              ExponentialCone{Float64} of dim: 3
              ... and 234 more
    Settings: ϵ_abs = 1.0e-04, ϵ_rel = 1.0e-04,
              ϵ_prim_inf = 1.0e-06, ϵ_dual_inf = 1.0e-04,
              ρ = 0.1, σ = 1.0e-6, α = 1.6,
              max_iter = 2500,
              scaling iter = 10 (on),
              check termination every 40 iter,
              check infeasibility every 40 iter,
              KKT system solver: QDLDL
    Setup Time: 1.88ms

    Iter:	Objective:	Primal Res:	Dual Res:	Rho:
    40	3.8685e+01	3.8745e-02	4.9828e-03	1.0000e-01
    80	4.9739e+01	7.6007e-03	7.7075e-04	1.0000e-01
    120	5.1537e+01	1.6183e-03	1.4459e-04	1.0000e-01
    160	5.1856e+01	3.5038e-04	2.8090e-05	1.0000e-01

    ------------------------------------------------------------------
    >>> Results
    Status: Solved
    Iterations: 160
    Optimal objective: 51.8557
    Runtime: 0.249s (248.56ms)

    28-element Array{Float64,1}:
      2.674439930406765
      1.7748860393946884
      2.934236553994507
     -4.053008143592888
     -3.3811679943697612
     -4.0530397674103735
      0.7775813452248421
     -1.0919224676820907
     -0.46310226688930994
     -0.48894009682277273
     -3.2952703524511726
      0.5638302900174557
     -1.8175402706656334
      ⋮
     -0.47307150885083393
      0.6345032449500374
     -1.1631145722989464
     -1.2247379129099638
     -0.09277151198530442
     -2.685527206587255
      0.4665466584002503
     -0.7660832134314224
      0.44944903637367295
     -1.190858736040457
     -0.9542098066640803
     -1.1918683708910005

Once we have solved the optimisation problem and obtained the parameter vector $\theta$, we can plot the decision boundary. This can be done by evaluating our model over a grid of points $(u,v)$ and then plotting the contour line where the function returns a probability of $p=0.5$.

```julia
# First we evaluate our model over a grid of points z = Model(u, v)
u = collect(range(-1., stop = 1.5, length = 50))
v = collect(range(-1., stop = 1.5, length = 50))
z = zeros(length(u), length(v));
for i = 1:length(u), j = 1:length(v)
    z[i, j] = dot(map_feature(u[i], v[j]), theta);
end

# visualize data with decision boundary
PyPlot.figure(2)
p1 = plot(x1[findall(x -> x == 1., y)], x2[findall(x -> x == 1., y)], marker = "+", color = "b", linestyle= "None", label="Accepted")
p2 = plot(x1[findall(x -> x != 1., y)], x2[findall(x -> x != 1., y)], marker = ".", color = "r", linestyle= "None", label="Rejected")
axis("equal")
PyPlot.grid(true)
PyPlot.legend()
xlabel("x1 - Microchip Test Score 1")
ylabel("x2 - Microchip Test Score 2")
# Now add the decision boundary as a contour plot
contour(u, v, z', [0.5])
nothing
```

## Solving the optimisation problem directly with COSMO
We can solve the problem directly in COSMO by using its modeling interface. The problem will have $nn = 5 n + n_\theta + 1$ variables. Let us define the cost function  $ \frac{1}{2}x^\top P x + q^\top x $:

```julia
nn = 5 * n + n_theta +  1
P = spzeros(nn, nn)
q = zeros(nn)
q[1] = μ
for i = 1:n
  q[1 + n_theta + (i - 1) * 5 + 1] = 1.
end
```

Next we define a function that creates the `COSMO.Constraints` for a given sample:

```julia
# the order of the variables
# v, thetas, [e1 t11 t12 s11 s12] [e2 t21 t22 s21 s22] ...
# for each sample create two exponential cone constraints,
# 1 nonnegatives constraint, 2 zeroset constraints
function add_log_regression_constraints!(constraint_list, x, y, n, sample_num)
  num_thetas = length(x)
  # 1st exponential cone constraint (zi - ei, s1, t1) in Kexp
  c_start = 1 + num_thetas + (sample_num - 1) * 5 + 1
  A = spzeros(3, n)
  A[1, c_start] = -1.
  y == 1. ? (a = -1) : (a = 1)
  for k = 1:num_thetas
    A[1, 2 + k - 1] = a * x[k]
  end
  A[2, c_start + 3] = 1.
  A[3, c_start + 1] = 1.
  b = zeros(3)
  push!(constraint_list, COSMO.Constraint(A, b, COSMO.ExponentialCone))

  # 2nd exponential cone constraint (-e, s2, t2)
  A = spzeros(3, n)
  A[1, c_start] = -1.
  A[2, c_start + 4] = 1.
  A[3, c_start + 2] = 1.
  b = zeros(3)
  push!(constraint_list, COSMO.Constraint(A, b, COSMO.ExponentialCone))

  # Nonnegatives constraint t1 + t2 <= 1
  A = spzeros(1, n)
  A[1, c_start + 1] = -1.
  A[1, c_start + 2] = -1.
  b = [1.]
  push!(constraint_list, COSMO.Constraint(A, b, COSMO.Nonnegatives))

  # ZeroSet constraint s1 == 1, s2 == 1
  A = spzeros(2, n)
  A[1, c_start + 3] = 1.
  A[2, c_start + 4] = 1.
  b = -1 * ones(2)
  push!(constraint_list, COSMO.Constraint(A, b, COSMO.ZeroSet))
end
```

    add_log_regression_constraints! (generic function with 1 method)

Now we can use this function to loop over the sample points and add the constraints to our constraint list:

```julia
constraint_list = Array{COSMO.Constraint{Float64}}(undef, 0)
for i = 1:n
  add_log_regression_constraints!(constraint_list, X[i, :], y[i], nn, i )
end
```

It remains to add a second order cone constraint for the regularisation term:

$\|\theta \|_2 \leq v$

```julia
push!(constraint_list, COSMO.Constraint(Matrix(1.0I, n_theta + 1, n_theta + 1), zeros(n_theta + 1), COSMO.SecondOrderCone, nn, 1:n_theta+1))
nothing
```

We can now create, assemble, and solve our `COSMO.Model`:

```julia
model = COSMO.Model()
assemble!(model, P, q, constraint_list, settings = COSMO.Settings(verbose=true))
res = COSMO.optimize!(model);
```

    ------------------------------------------------------------------
                 COSMO - A Quadratic Objective Conic Solver
                             Michael Garstka
                    University of Oxford, 2017 - 2019
    ------------------------------------------------------------------

    Problem:  x ∈ R^{619},
              constraints: A ∈ R^{1091x619} (4513 nnz), b ∈ R^{1091},
              matrix size to factor: 1710x1710 (2924100 elem, 10736 nnz)
    Sets:     ZeroSet{Float64} of dim: 236
              Nonnegatives{Float64} of dim: 118
              SecondOrderCone{Float64} of dim: 29
              ExponentialCone{Float64} of dim: 3
              ExponentialCone{Float64} of dim: 3
              ExponentialCone{Float64} of dim: 3
              ... and 234 more
    Settings: ϵ_abs = 1.0e-04, ϵ_rel = 1.0e-04,
              ϵ_prim_inf = 1.0e-06, ϵ_dual_inf = 1.0e-04,
              ρ = 0.1, σ = 1.0e-6, α = 1.6,
              max_iter = 2500,
              scaling iter = 10 (on),
              check termination every 40 iter,
              check infeasibility every 40 iter,
              KKT system solver: QDLDL
    Setup Time: 2.51ms

    Iter:	Objective:	Primal Res:	Dual Res:	Rho:
    40	3.8685e+01	3.8745e-02	4.9828e-03	1.0000e-01
    80	4.9739e+01	7.6007e-03	7.7075e-04	1.0000e-01
    120	5.1537e+01	1.6183e-03	1.4459e-04	1.0000e-01
    160	5.1856e+01	3.5038e-04	2.8090e-05	1.0000e-01

    ------------------------------------------------------------------
    >>> Results
    Status: Solved
    Iterations: 160
    Optimal objective: 51.8557
    Runtime: 0.25s (249.68ms)

Let us double check that we get the same $\theta$ as in the previous section:

```julia
using Test
theta_cosmo = res.x[2:2+n_theta-1]
@test norm(theta_cosmo - theta) < 1e-10
```

    Test Passed

```julia

```


```python
import numpy as np
import sympy as sp
import scipy as sc
from scipy.optimize import fsolve
from numpy import linalg as la
```

$PROBLEM$ $1$

$Part$ $A$

```python
A = np.array([[2,5,-3],[5,7,1], [-3,1,4]])
print(A)
print("\nSince it's 3x3 Matrix we have 3 degree characterstic polynomial")
λ = sp.symbols('λ')
display(Eq(-λ**3+13*λ**2-15*λ-139,0))
```

$Part$ $B$

```python
#Let's define the obove characterstic polynomial and compute it
def func(λ):
    return -λ**3+13*λ**2-15*λ-139

λ1=fsolve(func,1)
print("λ1 =",λ1)
λ2=fsolve(func,10)
print("λ2 =",λ2)
λ3=fsolve(func,5)
print("λ3 =",λ3)

```

$Part$ $C$

```python
eigVal,eigVect=la.eig(A)
print("Eigen Values\n",eigVal)
print("\nEigenvectors\n",eigVect)
```

$Part$ $D$

```python
#for a Matrix to be orthogonal Q^T=Q^-1
Q=eigVect
print(Q.T,"\n")
print(np.linalg.inv(Q))
print("\n Thus the EigenVectors are Orthogonal")
```

$PROBLEM$ $2$

$Part$ $A$

```python
M=np.array([[-1,8,2,10,45],[8,6,-11,-23,85],[2,-11,4,-3,-7],[10,-23,-3,2,2],[45,85,-7,2,2]])
for i in range(25):
 q,r=la.qr(M)
 M=r@q

print("\nEigenvalues by QR factorization\n")
print(np.rint(M))
```

    Eigenvalues by QR factorization

    [[105.  -6.   0.  -0.  -0.]
     [ -6. -91.   0.   0.   0.]
     [  0.   0. -24.   0.  -0.]
     [ -0.   0.   0.  18.  -0.]
     [  0.  -0.  -0.  -0.   5.]]

$Part$ $B$

```python
eigVal,eigVect=la.eig(M)
print("Eigen Values\n",np.around(eigVal,3))
print("\nEigenvectors\n",np.around(eigVect,2))

```

    Eigen Values
     [104.99  -91.557 -23.637  18.211   4.993]

    Eigenvectors
     [[-1.   -0.03 -0.    0.    0.  ]
     [ 0.03 -1.   -0.    0.    0.  ]
     [-0.    0.   -1.    0.   -0.  ]
     [ 0.   -0.    0.    1.    0.  ]
     [-0.    0.   -0.   -0.    1.  ]]

$Part$ $C$

```python
print(np.trace(M))
print(sum(eigVal))
```

    12.999999999999865
    12.999999999999819

$Part$ $D$

```python
print(la.det(M))
print(np.prod(eigVal))
```

    20660378.000000045
    20660378.00000005

$PROBLEM$ $3$

$Part$ $A$

Rewriting the Matrix as Eigen Problem

\begin{pmatrix}
\frac{k}{m_{1}}-w^{2} & -\frac{k}{m_{1}} & 0\\
 -\frac{k}{m_{2}}& \frac{2k}{m_{2}}-w^{2} & -\frac{k}{m_{2}} \\
 0 & -\frac{k}{m_{1}} & \frac{k}{m_{1}}-w^{2}
\end{pmatrix}

$Part$ $B$

$Substituting $ the $ values $ of $ k, $ m1, $ m2 $ in $ the $ equation

\begin{pmatrix}
3.06*10^{28}-\lambda & -3.06*10^{28} & 0\\
 -1.20*10^{29} & 2.41*10^{29}-\lambda & -1.20*10^{29} \\
 0 & -3.06*10^{28} & 3.06*10^{28}-\lambda
\end{pmatrix}

```python
V=np.array([[3.06*10**28,-3.06*10**28,0],[-1.20*10**29,2.41*10**29,-1.20*10**29],[0,-3.06*10**28,3.06*10**28]])
print(V)
eigValues,eigVectors=la.eig(V)
print("\nThe vibration Frequenices (w^2)1, (w^2)2, (w^2)3 are \n\n",eigValues)
```

    [[ 3.06e+28 -3.06e+28  0.00e+00]
     [-1.20e+29  2.41e+29 -1.20e+29]
     [ 0.00e+00 -3.06e+28  3.06e+28]]

    The vibration Frequenices (w^2)1, (w^2)2, (w^2)3 are

     [2.71487288e+29 3.06000000e+28 1.12712460e+26]

$Part$ $C$

```python
print("\nEigen Vectors\n\n",eigVectors)
```

    Eigen Vectors

     [[-1.25028831e-01 -7.07106781e-01  5.78059140e-01]
     [ 9.84243660e-01 -1.84988936e-17  5.75929909e-01]
     [-1.25028831e-01  7.07106781e-01  5.78059140e-01]]

$PROBLEM$ $4$

$Part$ $A$

```python
S=np.array([[20,8,-18],[8,28,12],[-18,12,4]])
# print(S)
eigVal,eigVect=la.eig(S)
print("\nPrincipal Stresses in Mega Pascals are\n\n",eigVal)
```

    Principal Stresses in Mega Pascals are

     [-12.79567553  31.67988775  33.11578778]

$Part$ $B$

```python
#To find the angles we need the normal(eigenvectors) vectors frist
#Also python normalizes the normal vectors automatically
#angles are measured from the +ve axis CCW direction.
N=eigVect
print("Normal Vectors to the Prinipal Plane\n",N)

print("\nThe Maximum principal stress is 33.11 MPa")
print("It's Unit Normal to plane is ", N[:,2])

print("\nX-axis in degrees ",np.degrees(np.arccos(N[0,2]))," or radians",np.arccos(N[0,2]))
print("Y-axis in degrees ",np.degrees(np.arccos(N[1,2]))," or radians",np.arccos(N[1,2]))
print("Z-axis in degrees ",np.degrees(np.arccos(N[2,2]))," or radians",np.arccos(N[2,2]))
```

    Normal Vectors to the Prinipal Plane
     [[-0.51481828  0.81232573  0.27402382]
     [ 0.33329183 -0.10484587  0.93697593]
     [-0.78985992 -0.57370224  0.21676496]]

    The Maximum principal stress is 33.11 MPa
    It's Unit Normal to plane is  [0.27402382 0.93697593 0.21676496]

    X-axis in degrees  74.09615133093509  or radians 1.2932218037807959
    Y-axis in degrees  20.450248042933367  or radians 0.35692416119871395
    Z-axis in degrees  77.48090601002706  or radians 1.3522969173032346

$Part$ $C$

```python
#similary the Minimum Princial stress is -12.79Mpa
#Also python normalizes the normal vectors automatically
#angles are measured from the +ve axis CCW direction.
N=eigVect
print("Normal Vectors to the Prinipal Plane\n",N)

print("\nThe Minimum principal stress is -12.79 MPa")
print("It's Unit Normal to plane is ", N[:,0])

print("\nX-axis in degrees ",np.degrees(np.arccos(N[0,0]))," or radians",np.arccos(N[0,0]))
print("Y-axis in degrees ",np.degrees(np.arccos(N[1,0]))," or radians",np.arccos(N[1,0]))
print("Z-axis in degrees ",np.degrees(np.arccos(N[2,0]))," or radians",np.arccos(N[2,0]))
```

    Normal Vectors to the Prinipal Plane
     [[-0.51481828  0.81232573  0.27402382]
     [ 0.33329183 -0.10484587  0.93697593]
     [-0.78985992 -0.57370224  0.21676496]]

    The Minimum principal stress is -12.79 MPa
    It's Unit Normal to plane is  [-0.51481828  0.33329183 -0.78985992]

    X-axis in degrees  120.9853092679676  or radians 2.111591993269645
    Y-axis in degrees  70.53130164099609  or radians 1.2310034393526612
    Z-axis in degrees  142.17242293287993  or radians 2.4813768857166476


```python
import sympy as sp
sp.init_printing(use_latex = True)
%matplotlib inline
```

```python
M_s, x, w, y, h, a, beta, t, nu = sp.symbols('M_s, x, w, y, h, a, beta, t, nu')
```

```python
H_x = M_s/(4*sp.pi) * (sp.log(((x+w)**2 + (y-h)**2)/((x+w)**2 + (y+h)**2)) - sp.log(((x-w)**2 + (y-h)**2)/((x-w)**2 + (y+h)**2)))
H_x
```

```python
H_y = M_s/(2*sp.pi) * (sp.atan((2*h*(x+w))/((x+w)**2 + y**2 - h**2)) - sp.atan((2*h*(x-w))/((x-w)**2 + y**2 - h**2)))
H_y
```

```python
H = sp.sqrt(H_x**2 + H_y**2)
H
```

```python
Hx = sp.diff(H, x)
HX = Hx.subs(y, x)
print(HX)
```

    (M_s**2*(2*(2*h*(-2*w - 2*x)*(w + x)/(-h**2 + x**2 + (w + x)**2)**2 + 2*h/(-h**2 + x**2 + (w + x)**2))/(4*h**2*(w + x)**2/(-h**2 + x**2 + (w + x)**2)**2 + 1) - 2*(2*h*(-w + x)*(2*w - 2*x)/(-h**2 + x**2 + (-w + x)**2)**2 + 2*h/(-h**2 + x**2 + (-w + x)**2))/(4*h**2*(-w + x)**2/(-h**2 + x**2 + (-w + x)**2)**2 + 1))*(-atan(2*h*(-w + x)/(-h**2 + x**2 + (-w + x)**2)) + atan(2*h*(w + x)/(-h**2 + x**2 + (w + x)**2)))/(8*pi**2) + M_s**2*(-2*((-2*w + 2*x)/((h + x)**2 + (-w + x)**2) + (2*w - 2*x)*((-h + x)**2 + (-w + x)**2)/((h + x)**2 + (-w + x)**2)**2)*((h + x)**2 + (-w + x)**2)/((-h + x)**2 + (-w + x)**2) + 2*((-2*w - 2*x)*((-h + x)**2 + (w + x)**2)/((h + x)**2 + (w + x)**2)**2 + (2*w + 2*x)/((h + x)**2 + (w + x)**2))*((h + x)**2 + (w + x)**2)/((-h + x)**2 + (w + x)**2))*(-log(((-h + x)**2 + (-w + x)**2)/((h + x)**2 + (-w + x)**2)) + log(((-h + x)**2 + (w + x)**2)/((h + x)**2 + (w + x)**2)))/(32*pi**2))/sqrt(M_s**2*(-log(((-h + x)**2 + (-w + x)**2)/((h + x)**2 + (-w + x)**2)) + log(((-h + x)**2 + (w + x)**2)/((h + x)**2 + (w + x)**2)))**2/(16*pi**2) + M_s**2*(-atan(2*h*(-w + x)/(-h**2 + x**2 + (-w + x)**2)) + atan(2*h*(w + x)/(-h**2 + x**2 + (w + x)**2)))**2/(4*pi**2))

```python
H1 = H.subs(x, 0)
H1
```

```python
H2 = H1.subs(y, 0)
H2
```

```python
H3 = H2.subs(h, 12.5e-6)
H3
```

```python
H4 = H3.subs(w, 25e-6)
H4
```

```python
H5 = H4.subs(M_s, 8.6e5)
H5
```

```python
H5.evalf()   #H5 = H0
```

```python
print(H)
```

    sqrt(M_s**2*(-log(((-h + y)**2 + (-w + x)**2)/((h + y)**2 + (-w + x)**2)) + log(((-h + y)**2 + (w + x)**2)/((h + y)**2 + (w + x)**2)))**2/(16*pi**2) + M_s**2*(-atan(2*h*(-w + x)/(-h**2 + y**2 + (-w + x)**2)) + atan(2*h*(w + x)/(-h**2 + y**2 + (w + x)**2)))**2/(4*pi**2))


# Setting the beta hyper-parameters

Suppose $\theta \sim \beta(\alpha_1,\alpha_2)$ and we believe that $E[\theta] = m$ and $P(l < \theta < u) = 0.95$. Write a program that can solve for $\alpha_1$ and $\alpha_2$ in terms of $m$, $l$, and $u$.

The mean for the beta distribution is well-known, so we have that

\begin{equation}
E[\theta] = m = \frac{\alpha_1}{\alpha_1 + \alpha_2}.
\end{equation}

Thus, this implies that

\begin{equation}
\alpha_2 = \alpha_1\frac{1-m}{m}.
\end{equation}

Now, as $\alpha_1$ increases the strength of our prior increases, so $P(l < \theta < u)$ is a monotonic function of $\alpha_1$. And so, we can solve for $\alpha_1$, and thereby, $\alpha_2$ with a binary search.

```python
from scipy import stats

m = 0.15
l = 0.05
u = 0.3

alpha_1_lower = 0.01
alpha_1_upper = 1000
alpha_1 = (alpha_1_lower + alpha_1_upper)/2
tol = 1e-15
while alpha_1_upper - alpha_1_lower >= 1e-9:
    alpha_2 = alpha_1*(1-m)/m
    density = stats.beta.cdf(u, a = alpha_1, b = alpha_2) - stats.beta.cdf(l, a = alpha_1, b = alpha_2)
    if density > 0.95:
        alpha_1_upper = alpha_1
    else:
        alpha_1_lower = alpha_1
    alpha_1 = (alpha_1_lower + alpha_1_upper)/2

print(alpha_1)
print(alpha_2)
```

    4.506062413888118
    25.534353681276208

The problem asks for $\alpha_1$ and $\alpha_2$ if $m = 0.15$, $l = 0.05$, and $u = 0.3$. We find that

\begin{align}
\alpha_1 &\approx 4.506062413888118 \\
\alpha_2 &\approx 25.534353681276208.
\end{align}


## BNH

Binh and Korn defined the following test problem in <cite data-cite="bnh"></cite> with 2 objectives and 2 constraints:

**Definition**

\begin{equation}
\newcommand{\boldx}{\mathbf{x}}
\begin{array}
\mbox{Minimize} & f_1(\boldx) = 4x_1^2 + 4x_2^2, \\
\mbox{Minimize} & f_2(\boldx) = (x_1-5)^2 + (x_2-5)^2,    \\
\mbox{subject to} & C_1(\boldx) \equiv (x_1-5)^2 + x_2^2 \leq 25, \\
& C_2(\boldx) \equiv (x_1-8)^2 + (x_2+3)^2 \geq 7.7, \\
& 0 \leq x_1 \leq 5, \\
& 0 \leq x_2 \leq 3.
\end{array}
\end{equation}

**Optimum**

The Pareto-optimal solutions are constituted by solutions
$x_1^{\ast}=x_2^{\ast} \in [0,3]$ and $x_1^{\ast} \in [3,5]$,
$x_2^{\ast}=3$. These solutions are marked by using bold
continuous
curves.  The addition of both constraints in the problem does not make any solution
in the unconstrained Pareto-optimal front infeasible.
Thus, constraints may not introduce any additional difficulty
in solving this problem.

**Plot**

```python
from pymoo.factory import get_problem
from pymoo.util.plotting import plot

problem = get_problem("bnh")
plot(problem.pareto_front(), no_fill=True)
```


```python
import pandas as pd
import numpy as np
import scipy.stats as stats
from statsmodels.stats.proportion import proportions_ztest
```

## Frequentist A/B testing - Example -  Comparing two proportions

A/B testing is essentially a simple randomized trial.

When someone visits a website, they are randomly directed to one of two different landing pages. The purpose is to determine which page has a better conversion rate.

The key principle is that after a large number of visitors, the groups of people who visited the two pages are completely comparable in respect of all characteristics (age, gender, location etc). Consequenly, we can compare the two groups and obtain an unbiased assessment of which page has a better conversoin rate.

Below, we can see that Landing Page B has a higher conversion rate but is it statistical significant?

```python
data = pd.DataFrame({
    'landing_page': ['A', 'B'],
    'not_converted': [4514, 4473],
    'converted': [486, 527],
    'conversion_rate':[486/(486 + 4514), 527/(527 + 4473)]
})
data

```

      landing_page
      not_converted
      converted
      conversion_rate

      0
      A
      4514
      486
      0.0972

      1
      B
      4473
      527
      0.1054

### Formulate hypothesis

Conversion rate can be thought of as the proportion of visitors that had an order. Thus, we have are comparing two proportions. Our hypothesis test becomes:

- **Null Hypothesis:** there is no difference in proportions $p_a - p_b = 0$
- **Alternative Hypothesis:** there is a difference in proportions $p_a - p_b \ne 0$

### Assumptions

**1) Sample Size**
* $n_a*\hat{p_a}=486\geq10$
* $n_a*(1-\hat{p_a})=4515\geq10$
* $n_b*\hat{p_b}=527\geq10$
* $n_b*(1-\hat{p_b})=4472\geq10$

**2) Random Sample**

By design, our experiment uses a random sample

### Test Statistic

A test statistic is a single metric that can be used to evaluate the null hypothesis. A standard way to obtain this metric is to compute the z-score. This measures how many standard errors is our observe sample mean below or above the population mean

$$ \begin{align} z = \frac{(\hat{p_a}-\hat{p_b}) - (p_a-p_b)}{SE(\hat{p_a}-\hat{p_b})} \end{align} $$

$\hat{p_a}-\hat{p_b}$: the sample difference in proportions
$p_a-p_b$: the population difference in proportions
$SE(p_a-p_b)$: the standard error of the sample difference in proportions

$$\begin{align*}
& \text{Standard error is defined} \\
\\
& SE(X)=\frac{Var(x)}{\sqrt{n_x}} \\
\\
\\ & \text{Variance and covariance are defined} \\
\\
& Var(X) = E[X^2] - E[X]^2 \\
& Cov(X, Y) = E[XY] - E[X]E[Y] \\
\\
\\ & \text{Difference in variance between X and Y is defined} \\
\\
& Var(X - Y) = E[(X - Y)(X - Y)] - E[X - Y]^2 \\
& Var(X - Y) = E[X^2 - 2XY + Y^2] - (E[x] - E[y])^2 \\
& Var(X - Y) = E[X^2] - 2E[XY] + E[Y^2] - E[x]^2 + 2E[x]E[y] - E[y]^2 \\
& Var(X - Y) = (E[X^2] - E[x]^2) + (E[Y^2] - E[y]^2) - 2(E[XY] - E[X]E[Y])\\
& Var(X - Y) = Var(X) + Var(Y) -2Cov(X, Y) \\
\\
\\ & \text{Groups are independent thereofore covariance is 0} \\
\\
& Var(X - Y) = Var(X) + Var(Y)\\
\\
\\ & \text{Variance of a binomial proportion} \\
\\
& Var(p_a) = p_a (1 - p_a) \\
\\
\\ & \text{Standard error of a binomial proportion} \\
\\
& SE(p_a) = \frac{ p_a (1 - p_a)}{n_a}
\\
\\ & \text{thus} \\
\\
& Var(p_a-p_b) = Var(p_a) + Var(p_b) \\
& Var(p_a-p_b) = p_a(1-p_a) + p_b(1-p_b) \\
& SE(p_a-p_b) = \sqrt{\frac{p_a(1-p_a)}{n_a} + \frac{p_b(1-p_b)}{n_b}}
\\
\\ & \text{Under the null: } p_a=p_b=p \\
\\
& SE(p_a-p_b) = \sqrt{p(1-p)(\frac{1}{n_a}+\frac{1}{n_b})}
\end{align*}$$

### P-Value and hypothesis test outcome

```python
def ztest_proportion_two_samples(success_a, size_a, success_b, size_b, one_sided=False):
    """
    A/B test for two proportions;
    given a success a trial size of group A and B compute
    its zscore and pvalue

    Parameters
    ----------
    success_a, success_b : int
        Number of successes in each group

    size_a, size_b : int
        Size, or number of observations in each group

    one_side: bool
        False if it is a two sided test

    Returns
    -------
    zscore : float
        test statistic for the two proportion z-test

    pvalue : float
        p-value for the two proportion z-test
    """
    proportion_a = success_a/size_a
    proportion_b = success_b/size_b

    propotion = (success_a+success_b)/(size_a+size_b)
    se = propotion*(1-propotion)*(1/size_a+1/size_b)
    se = np.sqrt(se)

    z = (proportion_a-proportion_b)/se
    p_value = 1-stats.norm.cdf(abs(z))
    p_value *= 2-one_sided # if not one_sided: p *= 2

    return f"z test statistic: {z}, p-value: {p_value}"

success_a=486
size_a=486+4514
success_b=527
size_b=527+4473

ztest_proportion_two_samples(
    success_a=success_a,
    size_a=size_a,
    success_b=success_b,
    size_b=size_b,
)
```

    'z test statistic: -1.3588507649479744, p-value: 0.17419388311717388'

Under the null that the conversion rate of the page A and page B are equal, we would observe this difference in conversion rate with a probability of 17.4%. Our threshold is typically set to 5% and thus the difference of the conversion rate we observe does not give us sufficient evidence to reject the null.

**We fail to reject the null**


# Laboratório 6: Pesca

### Referente ao capítulo 11

Suponha que uma população de peixes é introduzida em um tanque artificial ou em uma região de água com redes.  Seja $x(t)$ o nível de peixes escalado em $t$, com $x(0) = x_0 > 0$. Os peixes inicialmente são pequenos e tem massa média um valor quase nula: trateremos como $0$. Após, a massa média é uma função
$$
f_{massa}(t) = k\frac{t}{t+1},
$$
onde $k$ é o máximo de massa possivelmente atingido. Consideraremos $T$ suficientemente pequeno de forma que não haja reprodução de peixes. Seja $u(t)$ a taxa de colheita e $m$ a taxa de morte natural do peixe. Queremos maximizar a massa apanhada no intervalo, mas minimizando os custos envolvidos. Assim o problema é

$$
\max_u \int_0^T Ak\frac{t}{t+1}x(t)u(t) - u(t)^2 dt, A \ge 0
$$
$$
\text{sujeito a  }x'(t) = -(m + u(t))x(t), x(0) = x_0,
$$
$$
0 \le u(t) \le M,
$$
onde $M$ é o limite físico da colheita.

## Condições Necessárias

### Hamiltoniano

$$
H = Ak\frac{t}{t+1}x(t)u(t) - u(t)^2 - \lambda(t)\left(m + u(t)\right)x(t)
$$

### Equação adjunta

$$
\lambda '(t) = - Ak\frac{t}{t+1}u(t) + \lambda(t)\left(m + u(t)\right)
$$

### Condição de transversalidade

$$
\lambda(T) = 0
$$

### Condição de otimalidade

$$
H_u = Ak\frac{t}{t+1}x(t) - 2u(t) - \lambda(t)x(t)
$$

$$
H_u < 0 \implies u^*(t) = 0 \implies x(t)\left(Ak\frac{t}{t+1} - \lambda(t)\right) < 0
$$

$$
H_u = 0 \implies 0 \le u^*(t) = 0.5x(t)\left(Ak\frac{t}{t+1} - \lambda(t)\right) \le M
$$

$$
H_u > 0 \implies u^*(t) = M \implies 0.5x(t)\left(Ak\frac{t}{t+1} - \lambda(t)\right) > M
$$

Assim $u^*(t) = \min\left\{M, \max\left\{0, 0.5x(t)\left(Ak\frac{t}{t+1} - \lambda(t)\right)\right\}\right\}$

### Importanto as bibliotecas

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
import sympy as sp

import sys
sys.path.insert(0, '../pyscripts/')

from optimal_control_class import OptimalControl
```

### Usando a biblitoca sympy

```python
t_sp, x_sp,u_sp,lambda_sp, k_sp, A_sp, m_sp = sp.symbols('t x u lambda k A m')
H = A_sp*k_sp*(t_sp/(t_sp+1))*x_sp*u_sp - u_sp**2 - lambda_sp*(m_sp + u_sp)*x_sp
H
```

$\displaystyle \frac{A k t u x}{t + 1} - \lambda x \left(m + u\right) - u^{2}$

```python
print('H_x = {}'.format(sp.diff(H,x_sp)))
print('H_u = {}'.format(sp.diff(H,u_sp)))
print('H_lambda = {}'.format(sp.diff(H,lambda_sp)))
```

    H_x = A*k*t*u/(t + 1) - lambda*(m + u)
    H_u = A*k*t*x/(t + 1) - lambda*x - 2*u
    H_lambda = -x*(m + u)

Resolvendo para $H_u = 0$

```python
eq = sp.Eq(sp.diff(H,u_sp), 0)
sp.solve(eq,u_sp)
```

    [x*(A*k*t - lambda*t - lambda)/(2*(t + 1))]

Aqui podemos descrever as funções necessárias para a classe.

```python
parameters = {'A': None, 'k': None, 'm': None, 'M': None}

diff_state = lambda t, x, u, par: -x*(par['m'] + u)
diff_lambda = lambda t, x, u, lambda_, par: - par['A']*par['k']*t*u/(t + 1) + lambda_*(par['m'] + u)
update_u = lambda t, x, lambda_, par: np.minimum(par['M'], np.maximum(0, 0.5*x*(par['A']*par['k']*t - lambda_*t - lambda_)/(t + 1)))
```

## Aplicando a classe ao exemplo

Vamos fazer algumas exeperimentações. Sinta-se livre para variar os parâmetros. Nesse caso passaremos os limites como parâmetro do `solve`.

```python
problem = OptimalControl(diff_state, diff_lambda, update_u)
```

```python
x0 = 0.4
T = 10
parameters['A'] = 5
parameters['k'] = 10
parameters['m'] = 0.2
parameters['M'] = 1
```

```python
t,x,u,lambda_ = problem.solve(x0, T, parameters, bounds = [(0, parameters['M'])])
ax = problem.plotting(t,x,u,lambda_)
for i in range(3):
    ax[i].set_xlabel('Semanas')
plt.show()
```

A estratégia ótima nesse caso inicia em $0$ e logo aumenta muito rapidamente, com um declínio posterior suave. A população é praticamente extinta no período considerado. O limite superior não teve efeito, dado que foi bem alto. Por isso, podemos testar com outros valores.

```python
parameters['M'] = 0.4
t,x,u,lambda_ = problem.solve(x0, T, parameters, bounds = [(0, parameters['M'])])
ax = problem.plotting(t,x,u,lambda_)
for i in range(3):
    ax[i].set_xlabel('Semanas')
plt.show()
```

Sugerimos que experimente a variação dos outros parâmetros.

## Experimentação

```python
#N0 = 1
#T = 5
#parameters['r'] = 0.3
#parameters['a'] = 10
#parameters['delta'] = 0.4
#
#t,x,u,lambda_ = problem.solve(N0, T, parameters)
#roblem.plotting(t,x,u,lambda_)
```

### Este é o final do notebook


```python
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
from scipy import stats
```

# Monte Carlo integration

## Integrals and averates

In general, we want to estimate integrals of a function over a distribution using the following simple strategy:

- Draw a probability from the target probability distribution
- Multiply the probability with the function at that value
- Repeat and average

The trick is in knowing how to draw samples from the target probability distribution.

### A Simple Method: Rejection sampling

Suppose we want random samples from some distribution for which we can calculate the PDF at a point, but lack a direct way to generate random deviates from. One simple idea that is also used in MCMC is rejection sampling - first generate a sample from a distribution from which we can draw samples (e.g. uniform or normal), and then accept or reject this sample with some probability (see figure).

#### Example: Random samples from the unit circle using rejection sampling

```python
x = np.random.uniform(-1, 1, (10000, 2))
x = x[np.sum(x**2, axis=1) < 1]
plt.scatter(x[:, 0], x[:,1], s=1)
plt.axis('square')
pass
```

#### Example: Rejection sampling from uniform distribution

We want to draw samples from a Cauchy distribution restricted to (-4, 4). We could choose a more efficient sampling/proposal distribution than the uniform, but this is just to illustrate the concept.

```python
x = np.linspace(-4, 4)

df = 10
dist = stats.cauchy()
upper = dist.pdf(0)
```

```python
plt.plot(x, dist.pdf(x))
plt.axhline(upper, color='grey')
px = 1.0
plt.arrow(px,0,0,dist.pdf(1.0)-0.01, linewidth=1,
          head_width=0.2, head_length=0.01, fc='g', ec='g')
plt.arrow(px,upper,0,-(upper-dist.pdf(px)-0.01), linewidth=1,
          head_width=0.3, head_length=0.01, fc='r', ec='r')
plt.text(px+.25, 0.2, 'Reject', fontsize=16)
plt.text(px+.25, 0.01, 'Accept', fontsize=16)
plt.axis([-4,4,0,0.4])
plt.title('Rejection sampling concepts', fontsize=20)
pass
```

```python
n = 100000
# generate from sampling distribution
u = np.random.uniform(-4, 4, n)
# accept-reject criterion for each point in sampling distribution
r = np.random.uniform(0, upper, n)
# accepted points will come from target (Cauchy) distribution
v = u[r < dist.pdf(u)]

plt.plot(x, dist.pdf(x), linewidth=2)

# Plot scaled histogram
factor = dist.cdf(4) - dist.cdf(-4)
hist, bin_edges = np.histogram(v, bins=100, normed=True)
bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2.
plt.step(bin_centers, factor*hist, linewidth=2)

plt.axis([-4,4,0,0.4])
plt.title('Histogram of accepted samples', fontsize=20)
pass
```

## Simple Monte Carlo integration

The basic idea of Monte Carlo integration is very simple and only requires elementary statistics. Suppose we want to find the value of
$$
I = \int_a^b f(x) dx
$$
in some region with volume $V$. Monte Carlo integration estimates this integral by estimating the fraction of random points that fall below $f(x)$ multiplied by $V$.

In a statistical context, we use Monte Carlo integration to estimate the expectation
$$
E[g(X)] = \int_X g(x) p(x) dx
$$

with

$$
\bar{g_n} = \frac{1}{n} \sum_{i=1}^n g(x_i)
$$
where $x_i \sim p$ is a draw from the density $p$.

We can estimate the Monte Carlo variance of the approximation as
$$
v_n = \frac{1}{n^2} \sum_{o=1}^n (g(x_i) - \bar{g_n})^2)
$$

Also, from the Central Limit Theorem,

$$
\frac{\bar{g_n} - E[g(X)]}{\sqrt{v_n}} \sim \mathcal{N}(0, 1)
$$

The convergence of Monte Carlo integration is $\mathcal{0}(n^{1/2})$ and independent of the dimensionality. Hence Monte Carlo integration generally beats numerical integration for moderate- and high-dimensional integration since numerical integration (quadrature) converges as $\mathcal{0}(n^{d})$. Even for low dimensional problems, Monte Carlo integration may have an advantage when the volume to be integrated is concentrated in a very small region and we can use information from the distribution to draw samples more often in the region of importance.

An elementary, readable description of Monte Carlo integration and variance reduction techniques can be found [here](https://www.cs.dartmouth.edu/~wjarosz/publications/dissertation/appendixA.pdf).

## Intuition behind Monte Carlo integration

We want to find some integral

$$I = \int{f(x)} \, dx$$

Consider the expectation of a function $g(x)$ with respect to some distribution $p(x)$. By definition, we have

$$
E[g(x)] = \int{g(x) \, p(x) \, dx}
$$

If we choose $g(x) = f(x)/p(x)$, then we have

$$
\begin{align}
E[g(x)] &= \int{\frac{f(x}{p(x)} \, p(x) \, dx} \\
&= \int{f(x) dx} \\
&= I
\end{align}
$$

By the law of large numbers, the average converges on the expectation, so we have

$$
I \approx \bar{g_n} = \frac{1}{n} \sum_{i=1}^n g(x_i)
$$

If $f(x)$ is a proper integral (i.e. bounded), and $p(x)$ is the uniform distribution, then $g(x) = f(x)$ and this is known as ordinary Monte Carlo. If the integral of $f(x)$ is improper, then we need to use another distribution with the same support as $f(x)$.

**Example: Estimating $\pi$**

We have a function

$$
f(x, y) =
\begin{cases}
1 & \text{if}\ x^2 + y^2 \le 1 \\
0 & \text{otherwise}
\end{cases}
$$

whose integral is
$$
I = \int_{-1}^{1} \int_{-1}^{1} f(x,y) dx dy = \pi
$$

So a Monte Carlo estimate of $\pi$ is
$$
Q = 4 \sum_{i=1}^{N} f(x, y)
$$

if we sample $p$ from the standard uniform distribution in $\mathbb{R}^2$.

```python
from scipy import stats
```

```python
x = np.linspace(-3,3,100)
dist = stats.norm(0,1)
a = -2
b = 0
plt.plot(x, dist.pdf(x))
plt.fill_between(np.linspace(a,b,100), dist.pdf(np.linspace(a,b,100)), alpha=0.5)
plt.text(b+0.1, 0.1, 'p=%.4f' % (dist.cdf(b) - dist.cdf(a)), fontsize=14)
pass
```

#### Using quadrature

```python
from scipy.integrate import quad
```

```python
y, err = quad(dist.pdf, a, b)
y
```

    0.47724986805182085

#### Simple Monte Carlo integration

If we can sample directly from the target distribution $N(0,1)$

```python
n = 10000
x = dist.rvs(n)
np.sum((a < x) & (x < b))/n
```

    0.4738

If we cannot sample directly from the target distribution $N(0,1)$ but can evaluate it at any point.

Recall that $g(x) = \frac{f(x)}{p(x)}$. Since $p(x)$ is $U(a, b)$, $p(x) = \frac{1}{b-a}$. So we want to calculate

$$
\frac{1}{n} \sum_{i=1}^n (b-a) f(x)
$$

```python
n = 10000
x = np.random.uniform(a, b, n)
np.mean((b-a)*dist.pdf(x))
```

    0.48069595176138846

## Intuition for error rate

We will just work this out for a proper integral $f(x)$ defined in the unit cube and bounded by $|f(x)| \le 1$. Draw a random uniform vector $x$ in the unit cube. Then

$$
\begin{align}
E[f(x_i)] &= \int{f(x) p(x) dx} = I \\
\text{Var}[f(x_i)] &= \int{(f(x_i) - I )^2 p(x) \, dx} \\
&= \int{f(x)^2 \, p(x) \, dx} - 2I \int(f(x) \, p(x) \, dx + I^2 \int{p(x) \, dx} \\
&= \int{f(x)^2 \, p(x) \, dx}  + I^2 \\
& \le \int{f(x)^2 \, p(x) \, dx} \\
& \le \int{p(x) \, dx} = 1
\end{align}
$$

Now consider summing over many such IID draws $S_n = f(x_1) + f(x_2) + \cdots + f(x_n)$. We have

$$
\begin{align}
E[S_n] &= nI \\
\text{Var}[S_n] & \le n
\end{align}
$$

and as expected, we see that $I \approx S_n/n$. From Chebyshev's inequality,

$$
\begin{align}
P \left( \left| \frac{s_n}{n} - I \right| \ge \epsilon \right)  &=
P \left( \left| s_n - nI \right| \ge n \epsilon \right) & \le \frac{\text{Var}[s_n]}{n^2 \epsilon^2} & \le
\frac{1}{n \epsilon^2} = \delta
\end{align}
$$

Suppose we want 1% accuracy and 99% confidence - i.e. set $\epsilon = \delta = 0.01$. The above inequality tells us that we can achieve this with just $n = 1/(\delta \epsilon^2) = 1,000,000$ samples, regardless of the data dimensionality.

### Example

We want to estimate the following integral $\int_0^1 e^x dx$.

```python
x = np.linspace(0, 1, 100)
plt.plot(x, np.exp(x))
plt.xlim([0,1])
plt.ylim([0, np.exp(1)])
pass
```

#### Analytic solution

```python
from sympy import symbols, integrate, exp

x = symbols('x')
expr = integrate(exp(x), (x,0,1))
expr.evalf()
```

    1.71828182845905

#### Using quadrature

```python
from scipy import integrate

y, err = integrate.quad(exp, 0, 1)
y
```

    1.7182818284590453

#### Monte Carlo integration

```python
for n in 10**np.array([1,2,3,4,5,6,7,8]):
    x = np.random.uniform(0, 1, n)
    sol = np.mean(np.exp(x))
    print('%10d %.6f' % (n, sol))
```

            10 1.847075
           100 1.845910
          1000 1.731000
         10000 1.727204
        100000 1.719337
       1000000 1.718142
      10000000 1.718240
     100000000 1.718388

### Monitoring variance in Monte Carlo integration

We are often interested in knowing how many iterations it takes for Monte Carlo integration to "converge". To do this, we would like some estimate of the variance, and it is useful to inspect such plots. One simple way to get confidence intervals for the plot of Monte Carlo estimate against number of iterations is simply to do many such simulations.

For the example, we will try to estimate the function (again)

$$
f(x) = x \cos 71 x + \sin 13x, \ \  0 \le x \le 1
$$

```python
def f(x):
    return x * np.cos(71*x) + np.sin(13*x)
```

```python
x = np.linspace(0, 1, 100)
plt.plot(x, f(x))
pass
```

#### Single MC integration estimate

```python
n = 100
x = f(np.random.random(n))
y = 1.0/n * np.sum(x)
y
```

    0.03103616434230248

#### Using multiple independent sequences to monitor convergence

We vary the sample size from 1 to 100 and calculate the value of $y = \sum{x}/n$ for 1000 replicates. We then plot the 2.5th and 97.5th percentile of the 1000 values of $y$ to see how the variation in $y$ changes with sample size. The blue lines indicate the 2.5th and 97.5th percentiles, and the red line a sample path.

```python
n = 100
reps = 1000

x = f(np.random.random((n, reps)))
y = 1/np.arange(1, n+1)[:, None] * np.cumsum(x, axis=0)
upper, lower = np.percentile(y, [2.5, 97.5], axis=1)
```

```python
plt.plot(np.arange(1, n+1), y, c='grey', alpha=0.02)
plt.plot(np.arange(1, n+1), y[:, 0], c='red', linewidth=1);
plt.plot(np.arange(1, n+1), upper, 'b', np.arange(1, n+1), lower, 'b')
pass
```

#### Using bootstrap to monitor convergence

If it is too expensive to do 1000 replicates, we can use a bootstrap instead.

```python
xb = np.random.choice(x[:,0], (n, reps), replace=True)
yb = 1/np.arange(1, n+1)[:, None] * np.cumsum(xb, axis=0)
upper, lower = np.percentile(yb, [2.5, 97.5], axis=1)
```

```python
plt.plot(np.arange(1, n+1)[:, None], yb, c='grey', alpha=0.02)
plt.plot(np.arange(1, n+1), yb[:, 0], c='red', linewidth=1)
plt.plot(np.arange(1, n+1), upper, 'b', np.arange(1, n+1), lower, 'b')
pass
```

## Variance Reduction

With independent samples, the variance of the Monte Carlo estimate is

$$
\begin{align}
\text{Var}[\bar{g_n}] &= \text{Var} \left[ \frac{1}{N}\sum_{i=1}^{N} \frac{f(x_i)}{p(x_i)} \right] \\
&= \frac{1}{N^2} \sum_{i=1}^{N}  \text{Var} \left[ \frac{f(x_i)}{p(x_i)} \right] \\
&= \frac{1}{N^2} \sum_{i=1}^{N} \text{Var}[Y_i] \\
&= \frac{1}{N} \text{Var}[Y_i]
\end{align}
$$

where $Y_i = f(x_i)/p(x_i)$. In general, we want to make $\text{Var}[\bar{g_n}]$ as small as possible for the same number of samples. There are several variance reduction techniques (also colorfully known as Monte Carlo swindles) that have been described - we illustrate the change of variables and importance sampling techniques here.

### Change of variables

The Cauchy distribution is given by
$$
f(x) = \frac{1}{\pi (1 + x^2)}, \ \ -\infty \lt x \lt \infty
$$

Suppose we want to integrate the tail probability $P(X > 3)$ using Monte Carlo. One way to do this is to draw many samples form a Cauchy distribution, and count how many of them are greater than 3, but this is extremely inefficient.

#### Only 10% of samples will be used

```python
import scipy.stats as stats

h_true = 1 - stats.cauchy().cdf(3)
h_true
```

    0.10241638234956674

```python
n = 100

x = stats.cauchy().rvs(n)
h_mc = 1.0/n * np.sum(x > 3)
h_mc, np.abs(h_mc - h_true)/h_true
```

    (0.13, 0.26932817794994063)

#### A change of variables lets us use 100% of draws

We are trying to estimate the quantity

$$
\int_3^\infty \frac{1}{\pi (1 + x^2)} dx
$$

Using the substitution $y = 3/x$ (and a little algebra), we get

$$
\int_0^1 \frac{3}{\pi(9 + y^2)} dy
$$

Hence, a much more efficient MC estimator is

$$
\frac{1}{n} \sum_{i=1}^n \frac{3}{\pi(9 + y_i^2)}
$$

where $y_i \sim \mathcal{U}(0, 1)$.

```python
y = stats.uniform().rvs(n)
h_cv = 1.0/n * np.sum(3.0/(np.pi * (9 + y**2)))
h_cv, np.abs(h_cv - h_true)/h_true
```

    (0.10219440906830025, 0.002167361082027339)

### Importance sampling

Suppose we want to evaluate

$$
I = \int{h(x)\,p(x) \, dx}
$$

where $h(x)$ is some function and $p(x)$ is the PDF of $y$. If it is hard to sample directly from $p$, we can introduce a new density function $q(x)$ that is easy to sample from, and write

$$
I = \int{h(x)\, p(x)\, dx} = \int{h(x)\, \frac{p(x)}{q(x)} \, q(x) \, dx}
$$

In other words, we sample from $h(y)$ where $y \sim q$ and weight it by the likelihood ratio $\frac{p(y)}{q(y)}$, estimating the integral as

$$
\frac{1}{n}\sum_{i=1}^n \frac{p(y_i)}{q(y_i)} h(y_i)
$$

Sometimes, even if we can sample from $p$ directly, it is more efficient to use another distribution.

#### Example

Suppose we want to estimate the tail probability of $\mathcal{N}(0, 1)$ for $P(X > 5)$. Regular MC integration using samples from $\mathcal{N}(0, 1)$ is hopeless since nearly all samples will be rejected. However, we can use the exponential density truncated at 5 as the importance function and use importance sampling. Note that $h$ here is simply the identify function.

```python
x = np.linspace(4, 10, 100)
plt.plot(x, stats.expon(5).pdf(x))
plt.plot(x, stats.norm().pdf(x))
pass
```

#### Expected answer

We expect about 3 draws out of 10,000,000 from $\mathcal{N}(0, 1)$ to have a value greater than 5. Hence simply sampling from $\mathcal{N}(0, 1)$ is hopelessly inefficient for Monte Carlo integration.

```python
%precision 10
```

    '%.10f'

```python
v_true = 1 - stats.norm().cdf(5)
v_true
```

    2.866515719235352e-07

#### Using direct Monte Carlo integration

```python
n = 10000
y = stats.norm().rvs(n)
v_mc = 1.0/n * np.sum(y > 5)
# estimate and relative error
v_mc, np.abs(v_mc - v_true)/v_true
```

    (0.0, 1.0)

#### Using importance sampling

```python
n = 10000
y = stats.expon(loc=5).rvs(n)
v_is = 1.0/n * np.sum(stats.norm().pdf(y)/stats.expon(loc=5).pdf(y))
# estimate and relative error
v_is, np.abs(v_is- v_true)/v_true
```

    (2.8290057563382236e-07, 0.01308555981236137)

```python

```


# Chain Rules and Total Probability

In this section, we introduce techniques that use conditional probability to decompose events. The goal is to express unknown probabilities of events in terms of probabilities that we already know.

## Using Conditional Probability to Decompose Events: Part 1 -- Chain Rules

From the definition of conditional probability, we can write $P(A|B)$ as
\begin{align}
  P(A|B)&= \frac{P(A \cap B)}{P(B)}  \\
  \Rightarrow P(A \cap B)&= P(A|B)P(B),
\end{align}

and we can write $P(B|A)$ as

\begin{align}
  P(B|A)&=\frac{P(A \cap B)}{P(A)}  \\
  \Rightarrow P(A \cap B)&= P(B|A)P(A).
\end{align}

After manipulating the expressions as shown, we get two *different* formula for expressing $P(A \cap B)$. These are **chain rules** for the probability of the intersection of two events. Such rules are often used when:
* Two events are dependent on each other, but the relation is simple if the outcome of one of the experiments is known.
* The events are at two different point in a system, such as the input and output of a system.

**Example**

A simple example of the former is in card games. Two cards are drawn (without replacement) from a deck of 52 cards (without jokers). What is the probability that they are both Aces? Let $A_i$ be the event that the card on draw $i$ is an Ace. Then the most natural way to apply the chain rule is to write

$$
P(A_1 \cap A_2) = P(A_2 | A_1) P(A_1).
$$

The probability of getting an Ace in draw 1 is 4/52=1/13 because there are 4 Aces in the deck of 52 cards.  The probability of getting an Ace on the second draw *given that the first draw was an Ace* is 3/51 = 1/17 because after the first draw, there are 3 Aces left in the remaining deck of 51 cards.  Thus,

$$
P(A_1 \cap A_2) = P(A_2 | A_1) P(A_1) = \left( \frac{1}{17} \right) \left( \frac{1}{13} \right) = \frac{1}{221}
$$

As a check, we can compare with a solution using combinatorics. There are

$$
\binom{4}{2} = 6
$$
ways to choose the two Aces from the four total Aces. There are

$$
\binom{52}{2} = \frac{ 52!}{50! 2!} = 1326
$$
ways to choose two cards from 52. So,

$$
P( A_1 \cap A_2) = \frac{6}{1326} = \frac{1}{221},
$$
which matches our answer using conditional probability.

The solution using conditional probability is usually much more intuitive for learners who are new to probability, but being able to use both techniques is a powerful method for checking your work

**To be added: Question on probability of getting two face cards (JQK) on consecutive draws from a deck of cards. Question on getting defective computers when sitting down at random computers in a lab.**

**Example**

For the Magician's Coin problem, what is the probability of getting the Fair coin and it coming up heads on the first flip? This is an example of a system where there is an input that affects the future outputs. In this case, the input is the choice of coin. When we have such problems, we usually will need to decompose them in terms of the probabilities of the input and the conditional probabilities of the output given the input. Let $H_i$ denote the event that the coin comes up heads on flip $i$. Let $F$ be the event that the fair coin was chosen.  We are looking for $P(F \cap H_1)$, which we can write as

$$
P(F \cap H_1) = P(H_1 | F) P(F).
$$

If there is one Fair coin and one two-headed coin, $P(F) =1/2$.  Given  that the coin is Fair, $P(H_1|F) = 1/2$. So,

$$
P(F \cap H_1) = \left( \frac 1 2  \right) \left( \frac 1 2  \right) = \frac 1 4.
$$

Note that it is generally **not helpful to write the probability using the other form of the chain rule:**

$$
P(F \cap H_1) = P(F| H_1) P(H_1).
$$

We do not know $P(H_1)$ nor $P(F|H_1)$. Thus, although the expression is valid mathematically, it is not helpful in solving this problem because it depends on probabilities that cannot be easily inferred from the problem setup.

**To be added:Question on probability  of getting the Fair coin and heads on first two flips.**

The chain rule can be easily generalized to more than two events. The easiest way is to write probabilities in terms of conditional probabilities that are expressed as fractions (as in the definition of probability), such that the denumenator of one fraction cancels with the numerator of the next fraction to make sure the expression is not changes when it is rewritten. This will make more sense with an example for rewriting the probability of the intersection of 3 events ($A$, $B$, and $C$):
\begin{align}
  P(A \cap B \cap C) &= \frac{P(A \cap B \cap C)} {}  \cdot
  \frac{\hspace{4em} }{} \cdot \frac{ \hspace{3em}\mbox{    }}{} \\
  &\\
  &= \frac{P(A \cap B \cap C)} {P(B \cap C)}  \cdot
  \frac{P(B \cap C)}{\mbox{   }} \cdot \frac{ \hspace{3em}\mbox{    }}{} \\
    &= \frac{P(A \cap B \cap C)} {P(B \cap C)}  \cdot
  \frac{P(B \cap C)}{P(C)} \cdot \frac{ P(C) }{1} \\
  &= P(A|B \cap C)  P(B | C) P(C)
\end{align}

This decomposition assumes we know the probability of  $A$ given that $B$ and $C$ have occurred and we know the probability of $B$ given $C$. Such dependence occurs naturally in many systems, but the particular decomposition will depend on what we know about these probabilities. We could just have easily written
\begin{align}
  P(A \cap B \cap C)
  &= P(C|A \cap B)  P(B | A) P(A)
\end{align}

## Using Conditional Probability to Decompose Events: Part 2 --  Partitions, and Total Probability

We previously introduced the concept of partitions in {doc}`Partitions<../03-first-data/partitions>` as a way to take a collection of data and break it into separate, disjoint groups. Now, we are ready to apply this concept to events, which are sets of outcomes. In particular, we will most often partition the sample space, $S$:

````{panels}
DEFINITION
^^^
partition (of the sample space)
: A collection of sets $A_1, A_2, \ldots$ *partitions*
  the sample space $S$ iff

  \begin{align*}
  S &= \bigcup_i A_i,     \mbox{ and }
  A_i \cap A_j &= \emptyset, i \ne j.
  \end{align*}

````

$\left\{A_i \right\}$ is also said to be a *partition* of $S$. For example, the collection of disjoint sets $A_0 A_1, \ldots, A_7$ shown in {numref}`sample-space-partition` is a partition for $S$:

:::{figure-md} sample-space-partition

Example partition of the sample space.
:::

Now consider how we can use a partition $A_0, A_1, \ldots, A_{n-1}$ of $S$ to decompose any event $B \subseteq S$. In {numref}`event-partition1`, an example event $B$ is shown on top of our example partition. Note that we do not require the $B$ have a nonempty intersection with every partition event (i.e., in this example $B \cap A_0 = \emptyset$).

:::{figure-md} event-partition1

Example event $B$ superimposed on partition $A_0, A_1, \ldots, A_7$ of the sample space.
:::

Then we can use our partition to decompose $B$ into smaller subsets as

$$
B_i = B \cap A_i, ~~ i = 0, 1, \ldots, n-1,
$$
as shown in {numref}`event-partition2`.

:::{figure-md} event-partition2

Example event $B$ superimposed on partition $A_0, A_1, \ldots, A_7$ of the sample space.
:::

Note that $A_i \cap A_j = \emptyset$ also implies that

$$
B_i \cap B_j = (B \cap A_i) \cap (B \cap A_j) = B \cap (A_i \cap A_j) = B \cap \emptyset =\emptyset,
$$
and

$$
\bigcup_i B_i = \bigcup_i \left(A_i \cap B\right )  = B \cap \left( \bigcup_i A_i \right) = B \cap S = B.
$$
These two properties imply that $B_0, B_1, \ldots, B_{n-1}$ are a partition for $B$. If we want to express the probability of $B$, we can write

\begin{align}
P(B) & = P \left( \bigcup_i B_i \right) \\
& = \sum_i P \left(B_i \right) \\
&= \sum_i P\left( B \cap A_i\right)
\end{align}

Now suppose that we choose the partitioning events $\{A_i\}$ such that:
* We know the probabilities $P(A_i)$
* We know the conditional probabilities of the event $B$ given that $A_i$ occurred, $P(B|A_i)$.

Applying chain rule, we can write $P(B \cap A_i) = P(B|A_i)P(A_i)$ for each $i$. Putting this all together, we get the *Law of Total Probability*:

````{panels}
Law of Total Probability
^^^
Given an event $B \subseteq S$ and a partition of $S$ denoted by
$A_1, A_2, \ldots$,

$$
P(B) = \sum_i P(B|A_i)P(A_i).
$$
````

The law of total probability is often used in systems where there is either:
* random inputs and outputs, where the output is dependent on the input
* a *hidden state*, which is some random property of the system that is not directly observable.

Note that these are not mutually exclusive. For the Magician's coin, we can treat the choice of coin as the input to the system or as a hidden state. When the coin is flipped repeatedly, then it generally makes more sense to interpret the choice of coin as a hidden state because it is a property of the system that cannot be directly observed but that influences the outpus of the system.

When applying chain rule in such problems, the Law of Total Probability will generally use conditioning on
  the different possibilities of the input or the hidden state.

**Example: The Magician's Coin**

As before, a magician has a fair coin and a two-headed coin in her pocket. Let $H_i$ denote the event that the outcome of flip $i$ is heads. We can use total probability to answer more complicated questions regarding the probabilities of the outputs:

**(a)** $P(H_1)$

As mentioned above, we can condition on the hidden state, which is whether the coin is fair ($F$) or not ($\overline{F}$).

```{note}
One thing that is often confusing to learners of probability is determining what is actually a partition of $S$.  If you have a set of events that are both *mutually exclusive* and *one of those events must occur* (i.e., the events cover the sample space), then they are a partition.

In this case, $F$ and $\overline{F}$ are complements, so they are mutually exclusive. Moreover, either the coin is fair ($F$) or it is not ($\overline{F}$), so one of these events must occur. Therefore the events $F, \overline{F}$ partition $S$.
```

Applying the Law of Total Probability,

\begin{align}
P(H_1) &= P(H_1|F)P(F) + P(H_1| \overline{F}) P( \overline{F}) \\
&= \left( \frac 1 2 \right) \left( \frac 1 2 \right) + \biggl( 1\biggr) \left( \frac 1 2 \right) \\
&= \frac 3 4
\end{align}

**(b)** $P(H_1 \cap H_2)$

We can use the same partition as above to write:
\begin{align}
P(H_1 \cap H_2) &= P(H_1 \cap H_2 |F)P(F) + P(H_1\cap H_2| \overline{F}) P( \overline{F}).
\end{align}
However, now we need to know the probabilities $P(H_1 \cap H_2 |F)$ and $P(H_1\cap H_2| \overline{F})$. When the coin is fair the events $H_1$ and $H_2$ are conditionally independent, so $P(H_1 \cap H_2 |F) =
P(H_1|F)P(H_2|F)$. When the coin is two-headed, it always comes up heads, so $P(H_1\cap H_2| \overline{F})=1$. Then
\begin{align}
P(H_1 \cap H_2) &= P(H_1|F)P( H_2 |F)P(F) + (1) P( \overline{F}) \\
&= \left( \frac 1 2 \right) \left( \frac 1 2 \right) \left( \frac 1 2 \right)
+\biggl( 1\biggr) \left( \frac 1 2 \right) \\
&= \frac 5 8.
\end{align}

**(c)** $P(H_2 \vert H_1)$

By calculating this probability, we can assess whether getting heads on flip 2 is independent of getting heads on flip 1, and if now, how information about the value of the first flip changes the probabilities for the second flip.  We can directly apply the definition of conditional probability to calculate this from the answers to parts (a) and (b):

$$
P(H_2 \vert H_1) = \frac{ P(H_1 \cap H_2) } {P(H_1)} = \frac{5/8}{3/4} = \frac{5}{6}
$$

If we did not know $H_1$, then $P(H_2)= P(H_1)=3/4$ (you should verify this using the Law of Total Probability with $H_1$ as the hidden state). So knowing that heads occurred on the first flip increases the probability that heads will occur on the second flip.

If this surprises you (again), then just recall that we can anticipate this if we take it to extremes. What if I told you that heads occurred on the first 1000 flips? Then you would surely think that the magician had chosen the two-headed coin and expect the probability of getting heads on the 1001th flip to be close to 1. Then if heads is  observed on one flip, we should expect the probability of getting heads on second flip to increase.

In fact, after observing one heads, the probability of having chosen the two-headed coin should increase to more than 1/2. It does -- but we will need some new tools that we will develop in Chapter 7 before we can calculate the new probability.

**TO DO: Create numerical answer questions for** $P(H_1 \cap H_2 \cap H_3)$,  $P(H_3 \vert H_1 \cap H_2)$

**Example: Two Random Selections**

At the beginning of this Chapter, the following question was asked: A box contains 2 white balls and 1 black ball. I reach in the box and remove one ball. Without inspecting it, I place it in my pocket. I withdraw a second ball. What is the probability that the second ball is white?

This is a question that stumps many learners of probability, but the answer turns out to be simple. Let $W_i$ be the event that a white ball is chosen on draw $i$. We are asking about $W_2$. Then $P(W_2) = P(W_1) = 2/3$.  However, this is not intuitive because our brain tells us that the probabilities for the second draw must depend on what happened on the first draw -- which is unknown in this case.

We can formally show this using the Law of Total Probability. The conditioning event will be the unobserved result of the first draw. The partition is $W_1$, $\overline{W_1}$. Then

\begin{align}
P(W_2) &= P(W_2|W_1) P(W_1) + P(W_2 | \overline{W_1}) P(\overline{W_1})\\
\end{align}

If a white ball is drawn first ($W_1$), then there is one white ball and one black ball left, so $P(W_2|W_1) = 1/2$. If a black ball is drawn first ($\overline{W_1}$), then there are two white balls left, so $P(W_1|
\overline{W_1}) = 1$.  Then
\begin{align}
P(W_2) &= \left( \frac 1 2 \right) \left( \frac 2 3 \right) + \biggl( 1 \biggr)
\left( \frac 1 3 \right) \\
&= \frac 2 3,
\end{align}
which is equal to $P(W_1)$.  We can use similar math to show that $P(W_3)=2/3$, also. In the absence of information about what happened on previous draws, the probability of getting a white ball is equal to the original proportion of white balls in the box.

**Example: The Monty Hall Problem**

You are on a game show, and you’re given the choice of three doors:

* Behind one door is a car
* Behind the other doors are goats

You pick a door, and the host, who knows what’s behind the doors, opens another door, which he knows has a goat.

The host then offers you the option to switch doors. Does it matter if you switch? If switching changes your probability of getting the prize, what is the new probability?

Here is a simple solution. Let $W_i$ be the event that you are winning on choice $i$. Let $S$ be the event that you switch. We are interested in comparing $P(W_2|S)$ with $P(W_2 | \overline{S})$.

Consider $\overline{S}$ first. Because there are two goats and the host always shows you a goat, the fact that he reveals a goat to you does not change the probability that you have selected the car. Thus $P(W_2|\overline{S}) = P(W_1) = 1/3$.

Now consider $S$.  It may be tempting to think that either:
1. Switching doesn't matter because the host was always going to show you a goat, so your new choice is just as likely to be a car as it was before, or
2. After the host reveals a goat, there is one goat and one car, so the probability of choosing the car when you switch is 1/2.

Unfortunately, neither of these are correct! Let's condition on what happens on choice 1:

$$
P(W_2|S) = P(W_2|S \cap W_1) P(W_1|S) + P(W_2|S \cap \overline{W_1} ) P(\overline{W_1}|S).
$$

The probability of winning on choice 1 does not depend on whether you switch after that choice, so $P(W_1|S) = P(W_1) = 1/3$, and $P(\overline{W_1}|S) = P(\overline{W_1})= 2/3.

Now, consider what happens on the second choice. There are two possibilities:
* If you are winning on the first choice ($W_1$), then the two doors you have not chosen both contain goats. The host reveals one of the goats, and you will switch to the other goat. Thus, $P(W_2|W_1) =0$.
* If you are not winning on the first choice ($\overline{W_1}$), then one of the doors you have not chosen has a goat and the other has the car. The host shows the goat that is not behind your door. Then if you switch, you will switch to the door with the car. Thus, $P(W_1 | \overline{W_1}) = 1$.

Putting this all together,

\begin{align}
P(W_2|S) &= P(W_2|S \cap W_1) P(W_1|S) + P(W_2|S \cap \overline{W_1} ) P(\overline{W_1}|S) \\
&= \biggl( 0 \biggr) \left( \frac 1 3 \right)
+ \biggl( 1 \biggr) \left( \frac 2 3 \right) \\
&= \frac 2 3.
\end{align}

Why is the probability not 1/2 using the reasoning above about one car and one goat left? Because you do not randomly choose among the two doors. The host is revealing information when he reveals the goat because he cannot choose your door, and he cannot choose the door with the car. If you were to randomly choose between the two doors after the goat is revealed, the probability would be 1/2.

```python

```