$ \newcommand{\pd}[2]{ \frac{\partial #1}{\partial #2} }
\newcommand{\od}[2]{\frac{d #1}{d #2}}
\newcommand{\td}[2]{\frac{D #1}{D #2}}
\newcommand{\ab}[1]{\langle #1 \rangle}
\newcommand{\bss}[1]{\textsf{\textbf{#1}}}
\newcommand{\ol}{\overline}
\newcommand{\olx}[1]{\overline{#1}^x}
$

# Advection, Diffusion, and Conservation Laws

## Basin Scale Budgets

As an introduction to this somewhat technical topic, we will first examine the _basin scale_ budgets of _mass, heat, and salt_. This are usually a bit more intuitive to understand.

Consider the volume shown below. It shows the side view of an ocean basin (like the North Atlantic) bounded at the bottom by the sea floor, at the top by the ocean surface, and at the southern edge by an open boundary through which water can flow. The central question here is, _what controls the rate of change of total mass, heat content, and salt inside this volume_?

```python
from IPython.display import Image
Image('images/3D_Basin_Diagram.png')
```

### Mass Conservation

Let $M$ be the total mass of the ocean in this basin. The rate of change of mass is given by

$$ \od{M}{t} = F_m^{surf} + F_{m}^{side} $$

The two terms on the right represent the mass flux into the volume at the surface and the mass flux into the volume at the side boundary. (Note that the sign coventions we chose for the $F_m$'s was arbitrary: we definied them such that mass will increase if they are positive. This choice dictates the sign in what follows.) At this point the equation is a bit obvious, but let's write what each term is a bit more explicity.

The total mass is given by the integral of the density over the volume (denoted by $V$):

$$ M = \iiint_V \rho dV \ . $$

The mass flux at the surface is given by the integral over the surface of evaporation ($E$, measured in m/s), precipitation ($P$) and runoff ($R$), multiplied by the density of fresh water:

$$ F_m^{surf} = -\iint_{surf} \rho_{fw} (E - P - R) dA \ . $$

The mass flux at the open boundary is given by the area integral of the velocity $\mathbf{u}$ times the density, in the direction normal to the boundary:

$$ F_m^{side} = - \iint_{side} \rho \mathbf{u} \cdot \mathbf{dA} \ .$$

Here $\mathbf{dA}$ is the vector area element which points _out_ of the volume normal to the boundary.

This equation suggests several different possibilities:
- If E-P is _not_ in balance with the net E-P-R, then the total ocean mass in the basin has to increase or decrease.
- In steady state (i.e. time derivative is zero), the net evaporation minus precipitation must balance the mass inflow at the boundary.

It is very common in oceanography to take the mass of the ocean, or of a basin, as constant in time. Although the ocean mass does change slightly on various timescales (most notably through the formation and melting of terrestrial ice sheets), these changes are extremely slight compared to the total ocean mass. In this case, we find

$$ F_m^{surf} = - F_{m}^{side} $$

### Mass Conservation in Boussinesq Approximation

The Boussisesq approximation is that density variations $\delta \rho$ are very small compared to the background density $\rho_0$, i.e.

$$ \rho = \rho _0 + \delta \rho \ ; \ \ |\delta \rho| \ll \rho_0 \ . $$

This approximation simplifies conservations equations considerably. The mass can now be approximated as

$$ M \simeq \iiint_V \rho_0 dV = \rho_0 \iiint_V dV \ . $$

Since $\rho_0$ is a constant, we can divide it out of the mass equation and work with volume instead:

$$ V =  \iiint_V dV \ . $$

The mass equation becomes a volume equation:

$$ \begin{align}
\od{V}{t} =& \frac{F_m^{surf}}{\rho_0} + \frac{F_{m}^{side}}{\rho_0} \\
 =& F_v^{surf} + F_v^{side}
\end{align} $$

with

$$ F_v^{surf} = -\frac{\rho_{fw}}{\rho_0} \iint_{surf} (E - P - R) dA $$

and

$$ F_v^{side} = - \iint_{side} \mathbf{u} \cdot \mathbf{dA} \ . $$

### Heat Conservation

The total heat content of the basin is given by

$$ H = c_p^0 \iiint_V \rho  \Theta dV \ . $$

where $\Theta$ is the conservative temperature (proportional to potential enthalpy).
Some would say that that $\Theta$ should be measured in Kelvins. However, the [first law of thermodynamics](https://en.wikipedia.org/wiki/First_law_of_thermodynamics) only describes the _changes_ in heat content of a system; the absolute value of $H$ is meaningless.

The rate of change of heat content depends only on external fluxes into the system:

$$ \od{H}{t} = F_H^{surf} + F_H^{side} \ . $$

The surface flux is the air-sea flux $Q$ we discussed earlier, plus the advection of heat by the evaporating / precipitating water:

$$ F_H^{surf} = \iint_{surf} Q dA - c_p^0 \iint_{surf} \rho_{fw} \Theta (E - P - R) dA $$

where $Q$ is defined as positive for downward heat flux. The second term represents the advetion of heat into / out of the ocean surface by evaporating / precipitating water. Neglecting diffusive fluxes (for now), the flux through the side is

$$ F_H^{side} = -  c_p^0 \iint \mathbf{u} \rho \Theta \cdot \mathbf{dA} $$

#### Simultaneous Changes of Heat and Mass

It becomes difficult to reason about heat content when the mass of the system is also changing. For this reason, we will mostly talk about heat transport in situations where the total ocean mass, or the mass of an individual basin under consideration, is constant in time (i.e. $d M / d t = 0$).

If the mass of the system is changing, is may be useful to distinguish from the mean temperature of the body $\overline{\Theta}$ and a fluctuation from this mean $\Theta'$. Expanding $H$, we obtain:

$$
\od{H}{t} = \od{}{t} \left [ c_p^0 \iiint_V \rho (\overline{\Theta} + \Theta') dV \right ]
=  c_p^0 \od{}{t} \iiint_V \rho \Theta' dV + c_p^0 \overline{\Theta} \od{M}{t}\ .
$$

We make the same substitution in the surface flux

$$ F_H^{surf} = \iint_{surf} Q dA - c_p^0 \iint_{surf} \rho_{fw} \Theta'(E - P - R) dA -
c_p^0 \overline{\Theta} \iint_{surf} \rho_{fw} (E - P - R) $$

and the side flux

$$ F_H^{side} = -  c_p^0 \iint_{side} \mathbf{u} \rho \Theta' \cdot \mathbf{dA} -
c_p^0 \overline{\Theta} \iint_{side} \mathbf{u} \rho \cdot \mathbf{dA}$$

Summing all these, we can obtain a formula of the form

$$
\od{H'}{t} = F_H^{'surf} + F_H^{'side} + c_p^0 \overline{\Theta} (\od{M}{t} - F_m^{surf} - F_m^{side})
$$

Through mass conservation, the final term proprtional to $\overline{\Theta}$ cancels out to zero.
All that remains are the terms proportional to $\Theta'$, which we have defined as

\begin{align}
H' &= c_p^0 \od{}{t} \iiint_V \rho \Theta' dV \\
F_H^{'surf} &= \iint_{surf} Q dA - c_p^0 \iint_{surf} \rho_{fw} \Theta'(E - P - R) dA \\
F_H^{'side} &= -  c_p^0 \iint_{side} \mathbf{u} \rho \Theta' \cdot \mathbf{dA} \ .
\end{align}

This gives us a template to follow when trying to calculate heat budgets for bodies whose mass is changing: redefine the temperature as relative to the mean temperature.
This form of the budget also reveals something important: even with $dM/dt = 0$, the magnitude of the surface and side fluxes is indeterminate up to a constant.
We can add _any_ constant temperature to these components, and it will cancel out in the heat budget.
Attempting to break down the heat budget into a sum of different components can lead to spurious conclusions, since these individual components potentially depend on an arbitrary reference temperature.
We can avoid this indeterminacy only if:
- There are no net mass fluxes at all through any of the boundaries ($F_m^{surf} = F_m^{side} = 0$) OR
- We reference all our temperature measurements at all boundaries to the same constant and, using the same reference temperature, cancel the terms related to net mass flux.

### Heat Conservation in Boussinesq Approximation

Under the Boussinesq approximation,

$$ H = c_p^0 \rho_0 \iiint_V \Theta dV \ . $$

The fluxes simplify to

$$ F_H^{surf} = \iint_{surf} Q dA - c_p^0 \frac{ \rho_{fw} }{\rho_0} \iint_{surf} \Theta (E - P - R) dA $$

and

$$ F_H^{side} = -  c_p^0 \rho_0 \iint_{side} \mathbf{u} \Theta \cdot \mathbf{dA} \ . $$

Volume conservation ($d V / dt = 0$) accompanies heat conservation to render it independent of $\Theta_{ref}$.

### Salt and Freshwater Conservation

Since negligible salt is exchanged with the atmosphere (or sea ice), the basin-scale budget for salinity is

$$\mathcal{S} = \iiint_V \rho S dV $$

$$ \frac{d \mathcal{S}}{dt} = F_S^{side} $$

with

$$ F_S^{side} = -  \iint_{side} \mathbf{u} \rho S \cdot \mathbf{dA} \ . $$

In steady state this simply states that the net salt flux through a closed section is zero.

We can construct the freshwater balance by subtracting the mass balance. Assuming steady state, we find

$$ F_M^{side} - F_S^{side} = -F_M^{surf}. $$

This equation can be simplified to

$$ F_{fw}^{side} = - \iint_{side} \mathbf{u} \rho(1 - S) \cdot \mathbf{dA}
   = \iint_{surf} \rho_{fw} (E - P - R) dA $$

which represents the next freshwater flux through the basin. In the Boussinesq approximation, we have

$$ F_{fw}^{side} = - \rho_0 \iint_{side} \mathbf{u} (1 - S) \cdot \mathbf{dA}
   = \rho_{fw}  \iint_{surf}(E - P - R) dA $$

For a basin with a closed volume budget ($F_V^{side} = 0$), the first term in parenthesis vanishes, and we obtain

$$ F_{fw}^{side} = - F_S^{side} $$

i.e. the freshwater transport is equal and opposite to the salt transport.

### Layered Models

```python
Image('images/3D_Basin_Diagram_layers.png')
```

A very common framework for thinking about basin-scale budgets is the use of _layered models_.
By this we mean models where the quantities of interest---$\mathbf{u}$, $\Theta$, etc.---are represented as piecewise constant function.
In these cases, we can replace many of the integrals in the budget with simple sums.

#### Example: Two Layer Flow

As an example, let's develop a model very losely inspired by the North Atlantic: flow coming into the basin in layer 1 and out of the basin in layer two. Let's use the Boussinesq assumption and additionally assume steady-state volume ($dV/dt = 0$) and no net volume flux through the surface $F_v^{surf} = 0$. In this case, the volume budget just becomes

$$
F_v^{side} = - \iint_{side} \mathbf{u} \cdot \mathbf{dA} = 0
$$

Now let's divide the integral into two separate components:
- In layer 1, $\mathbf{u} \cdot \mathbf{dA} < 0$ (inflow)
- In layer 2, $\mathbf{u} \cdot \mathbf{dA} > 0$ (outflow)

We now separate the integral into two parts

\begin{align}
F_v^{side} &= - \iint_1 \mathbf{u} \cdot \mathbf{dA} - \iint_2 \mathbf{u} \cdot \mathbf{dA} \\
&= \psi_1 + \psi_2 = 0
\end{align}

Clearly the two components must be equal and opposite, i.e.
$$ \psi_1 = \psi \ , \ \ \psi_2 = -\psi $$

In this simple two-layer model, the quantity $\psi$ represents the strength of the overturning circulation (covered much more in-depth in {doc}`overturning_circulation`).
It is a volume flux, measured in m$^3$/s, or in Sv (1 Sv = 10$^6$ m$^3$/s).
Note that we did not need to specify the exact details of $\mathbf{u}$ on the boundary or the shape of the boundary.

Now we will look at heat transport by this circulation.
The advective heat transport through the side boundary is given by

$$ F_H^{side} = -  c_p^0 \rho_0 \iint_{side} \mathbf{u} \Theta \cdot \mathbf{dA} \ . $$

This is where the piecewise-constant approximations of the layered model come in.
_We will assume that the temperature is uniform in each layer on the boundary._
I.e., on the side, we have $\Theta = \Theta_1$ in layer 1 and $\Theta = \Theta_2$ in layer 2.
The heat transport becomes

\begin{align}
F_H^{side} &= - c_p^0 \rho_0 \Theta_1 \iint_1 \mathbf{u} \cdot \mathbf{dA} -
   c_p^0 \rho_0 \Theta_1 \iint_2 \mathbf{u} \cdot \mathbf{dA} \\
   &= c_p^0 \rho_0 ( \Theta_1 \psi_1  + \Theta_2 \psi_2 )
\end{align}

Recognizing that $\psi_1 = \psi$ and $\psi_2 = -\psi$, we obtain

$$
F_H^{side} = c_p^0 \rho_0 \psi ( \Theta_1  - \Theta_2 ) \ ,
$$

where  $\Delta \Theta = \Theta_1 - \Theta_2$ is the difference in temperature between the two layers.
This simple yet powerful formula expresses some important fundamental truths about how ocean heat transport works:

- Heat transport can clearly occur even when there is no net volume transport, due to correlations between the strength / direction of the flow and the water temperature.
- The transport is proportional to the strength of the circulation, as measured by $\psi$.
- The transport is proportional to the temperature _difference_ between the inflowing and outflowing water.

## Micro-scale View

The integral budgets above illustrate the concept of heat conservation at the basin scale. We now move down-scale to consider an _infintessimally small_ element of fluid. These differential equations are the most general and flexible way to describe the conservation of mass, heat, and salt.

## Material Derivative #

The "water parcel" is hypothetical, infinitesimally small fluid element. We imagine the fluid to be composed of an infinite continuum of such parcels, each following its own unique path.
A fluid parcel position at a given time $t$ is given by a three dimensional vector $\mathbf{x}(x,y,z,t)$. The instantaneous rate of change of the position of such a fluid element defines the fluid velocity:

$$
\mathbf{u}  = (u, v, w)\equiv \od{\mathbf{x}}{t}
$$

One way to do fluid mechanics is to keep track of all such fluid elements as they move around; this is called the _Lagrangian_ approach. In Lagrangian fluid mechanics, the rate of change of some fluid property $c$ is just $dc / dt$; it is implicit that we are following the water parcel along its path through space. As you might imagine, Lagrangian fluid mechanics is very difficult because there is actually an infinite number of such paths, and the trajectories quickly become extremely complex. This approach is also incompatible with the way we usually measure the ocean, which involves taking samples at a fixed point in space.

The alternative is the _Eulerian_ approach. Under this approach, the rate of change of a fluid parcel property is proportional to the _local_ rate of change of that property, plus a contribution from the fluid flow transporting that property around (called advection). We can see this mathemtically by using the chain rule:

$$ \od{}{t} c(\mathbf{x}, t) = \pd{c}{t} + \od{\mathbf{x}}{t} \cdot \nabla c
= \left ( \pd{}{t} + \mathbf{u} \cdot \nabla \right ) c \ . $$

The quantity in parentheses is called the _material derivative_. To distinguish it from an ordinary or partial derivative, we often denote it with a capital $D$:

$$ \td{}{t} = \pd{}{t} + \mathbf{u} \cdot \nabla  $$

It represents a rate of change following a fluid parcel, but in the Eulerian frame of reference.

### Mass Conservation: Continuity Equation

The mass budget of an infinitesimally small fluid parcel is

$$ \pd{\rho}{t} + \nabla \cdot (\rho \mathbf{u}) = 0 \ . $$

This equation is called the _continuity equation_.

Using some [vector calculus identities](https://en.wikipedia.org/wiki/Vector_calculus_identities), it can be rewritten as.

$$ \pd{\rho}{t} + \mathbf{u} \cdot \nabla \rho + \rho \nabla \cdot \mathbf{u} = 0  $$

or, using the definition of material derivative

$$ \td{\rho}{t} + \rho \nabla \cdot \mathbf{u} = 0  $$

### Continuity Equation in Boussinesq Approximation

In the Boussinesq approximation, fluctuations in density are much smaller than the background density. If we expand the continuity equation interms of background and fluctuations, we obtain

$$ \td{\rho_0}{t} +  \td{\delta \rho}{t} + \rho_0 \nabla \cdot \mathbf{u} + \delta \rho \nabla \cdot \mathbf{u} = 0 $$

The first terms is zero, because $\rho_0$ is constant. Of the remaining nonzero terms, only one has a $\rho_0$ in it. As a "first order" approximation, we find that

$$ \rho_0 \nabla \cdot \mathbf{u} \simeq 0  \ .$$

At this point, we are free to drop the $\rho_0$ factor to give the Boussinesq volume continuity equation:

$$ \nabla \cdot \mathbf{u} = 0  \ .$$

This is also called the incompressibility equation. In the Boussinesq approximation, volume conservation replaced mass conservation.

### Heat Conservation

The left-hand side of the heat conservation equation, representing the rate of change of the heat content of a fluid parcel is

$$ c_p^0 \left [ \pd{}{t}(\rho \Theta) + \nabla \cdot ( \rho \Theta \mathbf{u} ) \right ]$$

which we can rewrite using the continuity equation as

$$ c_p^0 \rho \td{\Theta}{t} \ . $$

The right hand side, representing all non-conservative effects, can be written as the sum of the convergence of a molecular diffusive flux $\mathbf{Q}_{diff}$ and a radiative flux $Q_{rad}$:

(Note we have assumed that the radiative flux is only in the vertical direction, a very reasonable approximation for large-scale processes.)

The diffusive flux is given by [Fick's law of diffusion](https://en.wikipedia.org/wiki/Fick%27s_laws_of_diffusion):

$$ \mathbf{Q}_{diff}  = - c_p^0 \rho \kappa_T \nabla T $$

where $T$ is the _in-situ_ temperature and $\kappa_T$ is the molectular diffusivity of heat. The air-sea heat flux acts as a boundary condition for the diffusive flux.

In the Boussinesq approximation, we replace the $\rho$'s above with $\rho_0$ to obtain:

$$ \td{\Theta}{t} =  \kappa_T \nabla^2 T  - \frac{1}{\rho_0 c_p^0}\pd{Q_{rad}}{z} \ .$$

One complictation of this equation is that molecular diffusion acts on $T$, not $\Theta$. However, we will mostly be thinking about turbulent, rather than molecular, diffusion, for which this is not an issue.

### Salt Conservation

Neglecting the exchange of salt with the surface, salinity is governed just by advection and diffusion

$$  \rho \td{S}{t} =  \nabla \cdot ( \rho \kappa_S \nabla S ) $$

which reduces to

$$ \td{S}{t} =   \kappa_S  \nabla^2 S  $$

in the Boussinesq approximation.

```python

```


# K-Means Clustering

# 1) Use the "Breast Cancer Wisconsin (Diagnostic) Data Set" from Kaggle to try and cluster types of cancer cells.

Here's the original dataset for your reference:

<https://www.kaggle.com/uciml/breast-cancer-wisconsin-data>

## This is a supervised learning dataset

(Because it has **labels** - The "diagnosis" column.)

```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA # You don't necessarily have to use this
from sklearn.cluster import KMeans # You don't necessarily have to use this
from sklearn.preprocessing import StandardScaler # You don't necessarily have to use this

df = pd.read_csv("https://raw.githubusercontent.com/ryanleeallred/datasets/master/Cancer_Cells.csv")
print(df.shape)
df.head()
```

    (569, 33)

      id
      diagnosis
      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst
      fractal_dimension_worst
      Unnamed: 32

      0
      842302
      M
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871
      1.0950
      0.9053
      8.589
      153.40
      0.006399
      0.04904
      0.05373
      0.01587
      0.03003
      0.006193
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890
      NaN

      1
      842517
      M
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667
      0.5435
      0.7339
      3.398
      74.08
      0.005225
      0.01308
      0.01860
      0.01340
      0.01389
      0.003532
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902
      NaN

      2
      84300903
      M
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999
      0.7456
      0.7869
      4.585
      94.03
      0.006150
      0.04006
      0.03832
      0.02058
      0.02250
      0.004571
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758
      NaN

      3
      84348301
      M
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744
      0.4956
      1.1560
      3.445
      27.23
      0.009110
      0.07458
      0.05661
      0.01867
      0.05963
      0.009208
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300
      NaN

      4
      84358402
      M
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883
      0.7572
      0.7813
      5.438
      94.44
      0.011490
      0.02461
      0.05688
      0.01885
      0.01756
      0.005115
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678
      NaN

## Now it's an unsupervised learning dataset

(Because we've removed the diagnosis label) - Use this version.

```
df = df.drop('diagnosis', axis=1)
df.head()
```

      id
      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst
      fractal_dimension_worst

      0
      842302
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871
      1.0950
      0.9053
      8.589
      153.40
      0.006399
      0.04904
      0.05373
      0.01587
      0.03003
      0.006193
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890

      1
      842517
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667
      0.5435
      0.7339
      3.398
      74.08
      0.005225
      0.01308
      0.01860
      0.01340
      0.01389
      0.003532
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902

      2
      84300903
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999
      0.7456
      0.7869
      4.585
      94.03
      0.006150
      0.04006
      0.03832
      0.02058
      0.02250
      0.004571
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758

      3
      84348301
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744
      0.4956
      1.1560
      3.445
      27.23
      0.009110
      0.07458
      0.05661
      0.01867
      0.05963
      0.009208
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300

      4
      84358402
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883
      0.7572
      0.7813
      5.438
      94.44
      0.011490
      0.02461
      0.05688
      0.01885
      0.01756
      0.005115
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678

## Let's do it!

- You might want to do some data exploration to see if you can find specific columns that will help you find distinct clusters of cells
- You might want to use the elbow method to decide on the number of clusters to use.

```
# Perform K-Means Clustering on the Dataset
df.dtypes
```

    id                           int64
    radius_mean                float64
    texture_mean               float64
    perimeter_mean             float64
    area_mean                  float64
    smoothness_mean            float64
    compactness_mean           float64
    concavity_mean             float64
    concave points_mean        float64
    symmetry_mean              float64
    fractal_dimension_mean     float64
    radius_se                  float64
    texture_se                 float64
    perimeter_se               float64
    area_se                    float64
    smoothness_se              float64
    compactness_se             float64
    concavity_se               float64
    concave points_se          float64
    symmetry_se                float64
    fractal_dimension_se       float64
    radius_worst               float64
    texture_worst              float64
    perimeter_worst            float64
    area_worst                 float64
    smoothness_worst           float64
    compactness_worst          float64
    concavity_worst            float64
    concave points_worst       float64
    symmetry_worst             float64
    fractal_dimension_worst    float64
    dtype: object

```
df = df.iloc[:, :-1]
df.head()
```

      id
      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst

      0
      842302
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871
      1.0950
      0.9053
      8.589
      153.40
      0.006399
      0.04904
      0.05373
      0.01587
      0.03003
      0.006193
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601

      1
      842517
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667
      0.5435
      0.7339
      3.398
      74.08
      0.005225
      0.01308
      0.01860
      0.01340
      0.01389
      0.003532
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750

      2
      84300903
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999
      0.7456
      0.7869
      4.585
      94.03
      0.006150
      0.04006
      0.03832
      0.02058
      0.02250
      0.004571
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613

      3
      84348301
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744
      0.4956
      1.1560
      3.445
      27.23
      0.009110
      0.07458
      0.05661
      0.01867
      0.05963
      0.009208
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638

      4
      84358402
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883
      0.7572
      0.7813
      5.438
      94.44
      0.011490
      0.02461
      0.05688
      0.01885
      0.01756
      0.005115
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364

```

```

```

```

```
# standardizing the data
from sklearn.preprocessing import StandardScaler
feature_cols=df.columns[1:]
feature_cols[:10]
```

    Index(['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean',
           'smoothness_mean', 'compactness_mean', 'concavity_mean',
           'concave points_mean', 'symmetry_mean', 'fractal_dimension_mean'],
          dtype='object')

```
x = df.loc[:, feature_cols].values
x = StandardScaler().fit_transform(x)
```

```
print('{:.2f}'.format(np.mean(x)))
print(np.std(x))
```

    0.00
    1.0

```
normal_cancer = pd.DataFrame(x,columns=feature_cols)
normal_cancer.head(5)
```

      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst

      0
      1.097064
      -2.073335
      1.269934
      0.984375
      1.568466
      3.283515
      2.652874
      2.532475
      2.217515
      2.255747
      2.489734
      -0.565265
      2.833031
      2.487578
      -0.214002
      1.316862
      0.724026
      0.660820
      1.148757
      0.907083
      1.886690
      -1.359293
      2.303601
      2.001237
      1.307686
      2.616665
      2.109526
      2.296076
      2.750622

      1
      1.829821
      -0.353632
      1.685955
      1.908708
      -0.826962
      -0.487072
      -0.023846
      0.548144
      0.001392
      -0.868652
      0.499255
      -0.876244
      0.263327
      0.742402
      -0.605351
      -0.692926
      -0.440780
      0.260162
      -0.805450
      -0.099444
      1.805927
      -0.369203
      1.535126
      1.890489
      -0.375612
      -0.430444
      -0.146749
      1.087084
      -0.243890

      2
      1.579888
      0.456187
      1.566503
      1.558884
      0.942210
      1.052926
      1.363478
      2.037231
      0.939685
      -0.398008
      1.228676
      -0.780083
      0.850928
      1.181336
      -0.297005
      0.814974
      0.213076
      1.424827
      0.237036
      0.293559
      1.511870
      -0.023974
      1.347475
      1.456285
      0.527407
      1.082932
      0.854974
      1.955000
      1.152255

      3
      -0.768909
      0.253732
      -0.592687
      -0.764464
      3.283553
      3.402909
      1.915897
      1.451707
      2.867383
      4.910919
      0.326373
      -0.110409
      0.286593
      -0.288378
      0.689702
      2.744280
      0.819518
      1.115007
      4.732680
      2.047511
      -0.281464
      0.133984
      -0.249939
      -0.550021
      3.394275
      3.893397
      1.989588
      2.175786
      6.046041

      4
      1.750297
      -1.151816
      1.776573
      1.826229
      0.280372
      0.539340
      1.371011
      1.428493
      -0.009560
      -0.562450
      1.270543
      -0.790244
      1.273189
      1.190357
      1.483067
      -0.048520
      0.828471
      1.144205
      -0.361092
      0.499328
      1.298575
      -1.466770
      1.338539
      1.220724
      0.220556
      -0.313395
      0.613179
      0.729259
      -0.868353

```
from sklearn.decomposition import PCA
pca_cancer = PCA(n_components=2)
pca_data = pca_cancer.fit_transform(x)
df_final = pd.DataFrame(data = pca_data, columns = ['comp_1', 'comp_2'])
df_final.head()
```

      comp_1
      comp_2

      0
      8.966336
      1.984413

      1
      2.463064
      -3.856484

      2
      5.777253
      -0.841503

      3
      6.283637
      9.494189

      4
      4.077171
      -1.544879

```
# variation
pca_cancer.explained_variance_ratio_
```

    array([0.45067848, 0.18239963])

```
# total variance
print('total variance explained: ', sum(pca_cancer.explained_variance_ratio_))
print('lost info: ', 1-sum(pca_cancer.explained_variance_ratio_))
```

    total variance explained:  0.6330781167216935
    lost info:  0.36692188327830655

```
# add back tumor type data
df_final['malignant']=df['diagnosis'].map({'M':1,'B':0})
df_final.sample(5)
```

      comp_1
      comp_2
      malignant

      477
      -3.103080
      -2.313723
      0

      524
      -3.368601
      0.982655
      0

      390
      -3.648998
      0.645817
      0

      416
      -2.679474
      2.615666
      0

      83
      5.012099
      1.128932
      1

```
plt.scatter(x= df_final['comp_1'],
            y= df_final['comp_2'],
            c= df_final['malignant']);
```

## Check you work:

This is something that in a truly unsupervised learning situation **WOULD NOT BE POSSIBLE**. But for educational purposes go back and grab the true diagnosis column (label) from the original dataset. Take your cluster labels and compare them to the original diagnosis column. You can make scatterplots for each to see how they compare or you can calculate a percent accuracy score like:
\begin{align}
\frac{\text{Num Correct Labels}}{\text{Num Total Observations}}
\end{align}

```
# Your Code Here
```

# 2) Perform PCA on your dataset first and *then* use k-means clustering.

- You need to standardize your data before PCA.
- First try clustering just on PC1 and PC2 so that you can make a scatterplot of your clustering.
- Then use use a scree plot to decide how many principal components to include in your clustering, and use however many principal components you need in order to retain 90% of the variation of the original dataset

```
df2 = pd.DataFrame(dict(x=df_final['comp_1'],
                   y=df_final['comp_2'],
                   label=df_final['malignant']))
df2.head()
```

      x
      y
      label

      0
      8.966336
      1.984413
      1

      1
      2.463064
      -3.856484
      1

      2
      5.777253
      -0.841503
      1

      3
      6.283637
      9.494189
      1

      4
      4.077171
      -1.544879
      1

```
plt.scatter(x=df2['x'], y=df2['y'], c=df2['label']);
```

```
points = df2.drop('label', axis=1)
points.head()
```

      x
      y

      0
      8.966336
      1.984413

      1
      2.463064
      -3.856484

      2
      5.777253
      -0.841503

      3
      6.283637
      9.494189

      4
      4.077171
      -1.544879

```
# Scatter plot of our label-less data (no more colors!)
plt.scatter(points.x, points.y);
```

## Check your work:

- Compare your PC1, PC2 clustering scatterplot to the clustering scatterplots you made on the raw data
- Calculate accuracy scores for both the PC1,PC2 Principal component clustering and the 90% of explained variance clustering.

How do your accuracy scores -when preprocessing the data with PCA- compare to the accuracy when simply clustering on the raw data?

# Stretch Goals:

- Study for the Sprint Challenge
- Work on your Data Storytelling Project

```

```


## Naïve Bayes – Additional Advice
This notebook guides you through multiple steps you can follow to create a naïve Bayes classifier. After following these steps you will still need to collate and move your code into the main assignment notebook file so that it meets the required format.

Read each step (including the maths!) carefully.

You can implement a naïve Bayes classifier without following this advice.

This notebook will not be graded and does not need to be submitted.

```python
import numpy as np

training_spam = np.loadtxt(open("training_spam.csv"), delimiter=",")
print("Shape of the spam training data set:", training_spam.shape)
print(training_spam)
```

    Shape of the spam training data set: (1000, 55)
    [[1. 0. 0. ... 0. 0. 0.]
     [0. 0. 1. ... 1. 0. 0.]
     [0. 0. 0. ... 1. 0. 0.]
     ...
     [0. 0. 0. ... 0. 0. 1.]
     [1. 1. 1. ... 1. 1. 0.]
     [1. 0. 0. ... 1. 1. 1.]]

## The model:  naïve Bayes
Your [naïve Bayes](https://en.wikipedia.org/wiki/Naive_Bayes_classifier) classifier will distinguish between two classes:

* $C = 1$ for spam messages
* $C = 0$ for ham messages

The classifier builds a model for the probability $p(C=c\ |\ \text{message})$ that a given message belongs to a certain class. A new message is then classified based on the Bayesian *maximum a posteriori* estimate
$\require{color}$
\begin{equation}
\hat{c} = \underset{c \in \{0,1\}}{\operatorname{argmax}} \  \textcolor{blue}{p(C=c\ |\ \text{message})}.
\end{equation}
Using Bayes' rule we can write

\begin{equation}
p(C=c\ |\ \text{message}) = \frac{p(\text{message}\ |\ C=c)p(C=c)}{p(\text{message}\ |\ C=1)p(C=1) + p(\text{message}\ |\ C=0)p(C=0)}.  \quad \quad
\end{equation}

The denominator is the same for both classes and we can thus drop it to get

\begin{equation}
\textcolor{blue}{p(C=c\ |\ \text{message})} \propto \textcolor{orange}{p(\text{message}\ |\ C=c)}\textcolor{green}{p(C=c)},
\end{equation}

where $\propto$ means "proportional to". The class priors $\textcolor{green}{p(C=c)}$ can be computed directly (you will do so in exercise A) but we need to further simplify $\textcolor{orange}{p(\text{message} \ |\ C=c)}$.

### Choice of the event model: *Multinomial* naïve Bayes

Different naïve Bayes models differ in their distributional assumptions about $\textcolor{orange}{p(\text{message}\ |\ C=c)}$. We represent a message using a **binary** [bag-of-words](https://en.wikipedia.org/wiki/Bag-of-words_model) model. Specifically, a message is represented as a set of $k$ keywords, that is, $message = (w_1, ..., w_k)$, where $w_i = 1$ if the  keyword $w_i$ appears in the message and $w_i = 0$ otherwise.

We assume that the $p(w_1, ..., w_k |\ C=c)$ follows a [multinomial distribution](https://en.wikipedia.org/wiki/Multinomial_distribution) for each class. Don't let the name scare you, this model simply assigns probabilities to different counts of events with multiple outcomes. So for example: "I roll a biased six-sided die six times, what is the probability that I get each side occurring exactly once" is a question that can be answered with a multinomial distribution. You don't need to understand all of the equations on the Wikipedia page.

Intuitively, the multinomial distribution assumes that the words of a message were "drawn" independently from a bag of $k$ different words. Depending on the class membership $c$, each keyword $w$ has a probability $\theta_{c, w}$ of being drawn. For example,

* $\theta_{spam, w}$ will have high value for $w \in \{$bank, transfer, buy,... $\}$.
* $\theta_{ham, w}$ will have high value for $w \in \{$paper, conference, proposal, experiment,... $\}$, if the training data was mostly gathered from emails of researchers.

Under these assumptions, the likelihood of a message, given that it belongs to class $c$, is then proportional to
\begin{equation}
\textcolor{orange}{p(\text{message}\ |\ C=c)} \propto \prod_{i = 1}^k  \left(\textcolor{brown}{\theta_{c, w_i}} \right)^{w_i}.
\end{equation}

The parameters $\textcolor{brown}{\theta_{c, w}}$ are estimated by counting the relative frequencies in the training data. Use **Laplace-smoothing** with $\alpha = 1$ (add-one smoothing), that is,
\begin{equation}
\textcolor{brown}{\theta_{c, w}} = \frac{n_{c, w} + \alpha}{n_{c} + k \alpha},
\end{equation}
where $n_{c, w}$ is the number of times the keyword $w$ appears in messages of class $c$ in the training set and $n_{c}$ is the total count of keywords for all messages of class $c$, that is, $n_{c} = \sum_w n_{c, w}$.

We are now finally able to rewrite the *maximum a posteriori* estimate in a form that is easy to compute:
\begin{equation}
\hat{c} = \underset{c \in \{0,1\}}{\operatorname{argmax}} \ \left[ \textcolor{green}{p(C=c)}   \prod_{i = 1}^k  \left(\textcolor{brown}{\theta_{c, w_i}} \right)^{w_i}\right].
\end{equation}

#### Increasing numerical stability
We can increase the numerical stability of the algorithm by taking logarithms of the posterior distributions, that is,
\begin{equation}
\hat{c} = \underset{c \in \{0,1\}}{\operatorname{argmax}} \ \log \left( \textcolor{green}{p(C=c)}   \prod_{i = 1}^k  \left(\textcolor{brown}{\theta_{c, w_i}}\right)^{w_i} \right) \\
 = \underset{c \in \{0,1\}}{\operatorname{argmax}} \ \left[ \log( \textcolor{green}{p(C=c)}) + \sum_{i = 1}^k w_i \ \log \left(\textcolor{brown}{\theta_{c, w_i}} \right) \right].
\end{equation}

## Part A: Estimate class priors

Define a function called `estimate_log_class_priors()` that takes as input a data set with binary response variable (0s and 1s) in the left-most column and returns a numpy array containing the **the logarithm** of the empirical class priors $\textcolor{green}{p(C=c)}$ for $c \in \{0, 1\}$.

```python
def estimate_log_class_priors(data):
    """
    Given a data set with binary response variable (0s and 1s) in the
    left-most column, calculate the logarithm of the empirical class priors,
    that is, the logarithm of the proportions of 0s and 1s:
        log(p(C=0)) and log(p(C=1))

    :param data: a two-dimensional numpy-array with shape = [n_samples, 1 + n_features]
                 the first column contains the binary response (coded as 0s and 1s).

    :return log_class_priors: a numpy array of length two
    """
    ### YOUR CODE HERE...
    return log_class_priors
```

```python
# You can use this cell to check whether the returned objects of your function are of the right data type.
log_class_priors = estimate_log_class_priors(training_spam)
print("result", log_class_priors)

# Check length
assert(len(log_class_priors) == 2)

# Check whether the returned object is a numpy.ndarray
assert(isinstance(log_class_priors, np.ndarray))

# Check wehther the values of this numpy.array are floats.
assert(log_class_priors.dtype == float)

# Check wehther the values are both negative (the logarithm of a probability 0 < p < 1 should be negative).
assert(np.all(log_class_priors < 0))
```

## Part B: Estimate class-conditional likelihoods
Define a function called `estimate_log_class_conditional_likelihoods()` that takes as input a data set with binary response variable (0s and 1s) in the left-most column and returns **the logarithm** of the empirical class-conditional likelihoods $\log \left(\textcolor{brown}{\theta_{c, w_i}} \right)$ for all words $w_i$ and both classes ($c \in {0, 1}$). These parameters should be returned in a two-dimensional numpy-array with shape = `[num_classes, num_features]`.

Assume a multinomial event model and use Laplace smoothing with $\alpha = 1$.

Hint: many `numpy`-functions contain an `axis` argument. If you specify `axis=0`, you can perform column-wise (that is, feature-wise!) computations.

```python
def estimate_log_class_conditional_likelihoods(data, alpha=1.0):
    """
    Given a data set with binary response variable (0s and 1s) in the
    left-most column and binary features (words), calculate the empirical
    class-conditional likelihoods, that is,
    log(P(w_i | c)) for all features w_i and both classes (c in {0, 1}).

    Assume a multinomial feature distribution and use Laplace smoothing
    if alpha > 0.

    :param data: a two-dimensional numpy-array with shape = [n_samples, 1 + n_features]

    :return theta:
        a numpy array of shape = [2, n_features]. theta[j, i] corresponds to the
        logarithm of the probability of feature i appearing in a sample belonging
        to class j.
    """
    ### YOUR CODE HERE...
    return theta

```

```python
# You can use this cell to check whether the returned objects of your function are of the right data type.
log_class_conditional_likelihoods = estimate_log_class_conditional_likelihoods(training_spam, alpha=1.0)
print(log_class_conditional_likelihoods)

# Check data type(s)
assert(isinstance(log_class_conditional_likelihoods, np.ndarray))

# Check shape of numpy array
assert(log_class_conditional_likelihoods.shape == (2, 54))

# Check data type of array elements
assert(log_class_conditional_likelihoods.dtype == float)
```

## Part  C: Classify e-mails

Having calculated the log class priors and the log class-conditional likelihoods for a given training set, define a function called `predict()`that takes a data set of new messages as input and predicts for each message whether it is spam or not. Note that the input should **not** contain a response variable.

```python
def predict(new_data, log_class_priors, log_class_conditional_likelihoods):
    """
    Given a new data set with binary features, predict the corresponding
    response for each instance (row) of the new_data set.

    :param new_data: a two-dimensional numpy-array with shape = [n_test_samples, n_features].
    :param log_class_priors: a numpy array of length 2.
    :param log_class_conditional_likelihoods: a numpy array of shape = [2, n_features].
        theta[j, i] corresponds to the logarithm of the probability of feature i appearing
        in a sample belonging to class j.
    :return class_predictions: a numpy array containing the class predictions for each row
        of new_data.
    """
    ### YOUR CODE HERE...
    return class_predictions

```

```python
# You can use this cell to check whether the returned objects of your function are of the right data type.
class_predictions = predict(training_spam[:, 1:], log_class_priors, log_class_conditional_likelihoods)

# Check data type(s)
assert(isinstance(class_predictions, np.ndarray))

# Check shape of numpy array
assert(class_predictions.shape == (1000,))

# Check data type of array elements
assert(np.all(np.logical_or(class_predictions == 0, class_predictions == 1)))
```

Now test your `predict` function by classifying messages. You can do this to the *training* data, but you should also try it on the *testing* data.

```python
# Check accuracy
true_classes = training_spam[:, 0]
training_set_accuracy = np.mean(np.equal(class_predictions, true_classes))
print(f"Accuracy on the training set: {training_set_accuracy}")
```

Once you are done, you can move the code into the main assignment notebook.

One way to do this is to follow the rough structure of the class that already exists in that notebook. You can use the `train` method to pass in the data and perform all of the steps before the prediction. You should store data in instance variables, e.g. `self.log_class_priors` and `self.log_class_conditional_likelihoods`. This means that then you can set up the `predict` method to match the one above without needing to pass in the additional variables. **Important:** the predict method must only take a single variable as a parameter (the one called `new_data`) in the skeleton code above.


```python
import numpy as np
import sympy as sp
import pandas as pd
import math
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

data=pd.read_csv('C:\\Users\\Utsav\\Desktop\\pulsar_prediction\\pulsar_stars.csv', sep=',',header=0)

pulsarData=data.values
pulsarData=np.array(pulsarData)
```

```python
#split the data

train, test = train_test_split(data, test_size=0.2)
pulsar=np.array(train)
train=np.array(train)
test=np.array(test)

print(len(test))
print(len(train))
#print(train)
```

    3580
    14318

```python
#copy labels of test & train data elsewhere
y_train=np.zeros(len(train))
y_test=np.zeros(len(test))
for i in range(len(test)):
    y_test[i]=test[i][8]
    test[i][8]=-1
#replace all test and train labels by -1
for i in range(len(train)):
    y_train[i]=train[i][8]
    train[i][8]=-1

```

```python

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5)
```

```python

knn.fit(train, y_train)
y_pred = knn.predict(test)

#calculate score (without scaling)
score=0
for i in range(len(test)):
    if y_pred[i]==y_test[i]:
        score=score+1
score=score*100/len(test)

print("Accuracy = " + "%.2f" % score + "%")

#calculate no of flase negatives
score=0
for i in range(len(test)):
    if y_pred[i]==0 and y_test[i]==1:
        score=score+1
#score=score*100/len(test)

print("No. of false negatives = %d" % score)

```

    Accuracy = 96.96%
    No. of false negatives = 77

```python

from sklearn.neighbors import KNeighborsClassifier

knn1 = KNeighborsClassifier(n_neighbors=5)
```

```python

# kNN with feature scaling
from sklearn.preprocessing import MaxAbsScaler

scaler=MaxAbsScaler()

train_scaled=scaler.fit_transform(train)
test_scaled=scaler.fit_transform(test)

knn1.fit(train_scaled, y_train)
y_pred = knn1.predict(test_scaled)

#calculate score (with scaling)
score=0
for i in range(len(test)):
    if y_pred[i]==y_test[i]:
        score=score+1
score=score*100/len(test)

print("Accuracy = " + "%.2f" % score + "%")

#calculate no of flase negatives
score=0
for i in range(len(test)):
    if y_pred[i]==0 and y_test[i]==1:
        score=score+1
#score=score*100/len(test)

print("No. of false negatives = %d" % score)
```

    Accuracy = 97.71%
    No. of false negatives = 54

```python
%%time
#implementation without sklearn

#without scaling

def md(arr1, arr2):
    md = 0
    for i in range(8):
        md = md + abs(arr1[i]-arr2[i])

    return md

for i in range(len(test)):
    dist = np.zeros(len(train))
    label = np.zeros(len(train))
    for j in range(len(train)):
        dist[j]=md(train[j], test[i])
        label[j]=y_train[j]

    l1 = np.argmin(dist)
    dist[l1] = 999999999999
    l1 = label[l1]
    l2 = np.argmin(dist)
    dist[l2] = 999999999999
    l2 = label[l2]
    l3 = np.argmin(dist)
    dist[l3] = 999999999999
    l3 = label[l3]

    knn = l1+l2+l3
    if(knn == 0):
        y_pred[i]=0
    if(knn == 1):
        y_pred[i]=0
    if(knn == 2):
        y_pred[i]=1
    if(knn == 3):
        y_pred[i]=1

score=0
for l in range(len(test)):
    if y_pred[l]==y_test[l]:
        score=score+1
score=score*100/len(test)
print("Accuracy = " + "%.2f" % score + "%")
```

    Accuracy = 97.23%
    Wall time: 5min 12s

```python
%%time
#implementation without sklearn
#with scaling

from sklearn.preprocessing import MaxAbsScaler

scaler=MaxAbsScaler()

train_scaled=scaler.fit_transform(train)
test_scaled=scaler.fit_transform(test)

"""
def md(arr1, arr2):
    md = 0
    for i in range(8):
        md = md + abs(arr1[i]-arr2[i])

    return md
"""

def ed(arr1, arr2):
    ed = 0
    for i in range(8):
        ed = ed + (arr1[i]-arr2[i])**2

    return math.sqrt(ed)

for i in range(len(test_scaled)):
    dist = np.zeros(len(train_scaled))
    label = np.zeros(len(train_scaled))
    for j in range(len(train_scaled)):
        dist[j]=ed(train_scaled[j], test_scaled[i])
        label[j]=y_train[j]

    l1 = np.argmin(dist)
    dist[l1] = 999999999999
    l1 = label[l1]
    l2 = np.argmin(dist)
    dist[l2] = 999999999999
    l2 = label[l2]
    l3 = np.argmin(dist)
    dist[l3] = 999999999999
    l3 = label[l3]

    knn = l1+l2+l3
    if(knn == 0):
        y_pred[i]=0
    if(knn == 1):
        y_pred[i]=0
    if(knn == 2):
        y_pred[i]=1
    if(knn == 3):
        y_pred[i]=1

score=0
for l in range(len(test_scaled)):
    if y_pred[l]==y_test[l]:
        score=score+1
score=score*100/len(test_scaled)
print("Accuracy = " + "%.2f" % score + "%")

#calculate no of flase negatives
score=0
pos=0
for i in range(len(test)):
    if y_pred[i]==0 and y_test[i]==1:
        score=score+1
    if y_test[i]==1:
        pos=pos+1

#score=score*100/len(test)
percent = score/len(test)

print("No of false negatives = %d" % score)
print("No of positives = %d" % pos)

```

    Accuracy = 97.54%
    Wall time: 7min 44s

```python
%%time
#kNN without feature scaling
#algorithm = bruteforce
#runnung on train data
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import MaxAbsScaler

scaler=MaxAbsScaler()

train_scaled=scaler.fit_transform(train)
test_scaled=scaler.fit_transform(test)

knn = KNeighborsClassifier(n_neighbors=5, algorithm='brute')
knn.fit(train_scaled, y_train)
y_pred = knn.predict(train_scaled)

score=0
for l in range(len(train_scaled)):
    if y_pred[l]==y_train[l]:
        score=score+1
score=score*100/len(train_scaled)
print("Accuracy = " + "%.2f" % score + "%")
```

    Accuracy = 98.27%
    Wall time: 8.77 s

```python
%%time
#implementation without sklearn
#with scaling

from sklearn.preprocessing import MaxAbsScaler

scaler=MaxAbsScaler()

train_scaled=scaler.fit_transform(train)
test_scaled=scaler.fit_transform(test)

def md(arr1, arr2):
    md = 0
    for i in range(8):
        md = md + abs(arr1[i]-arr2[i])

    return md

"""
def ed(arr1, arr2):
    ed = 0
    for i in range(8):
        ed = ed + (arr1[i]-arr2[i])**2

    return math.sqrt(ed)
"""

for i in range(len(test_scaled)):
    dist = np.zeros(len(train_scaled))
    label = np.zeros(len(train_scaled))
    for j in range(len(train_scaled)):
        dist[j]=md(train_scaled[j], test_scaled[i])
        label[j]=y_train[j]

    l1 = np.argmin(dist)
    dist[l1] = 999999999999
    l1 = label[l1]
    l2 = np.argmin(dist)
    dist[l2] = 999999999999
    l2 = label[l2]
    l3 = np.argmin(dist)
    dist[l3] = 999999999999
    l3 = label[l3]

    knn = l1+l2+l3
    if(knn == 0):
        y_pred[i]=0
    if(knn == 1):
        y_pred[i]=1
    if(knn == 2):
        y_pred[i]=1
    if(knn == 3):
        y_pred[i]=1

score=0
for l in range(len(test_scaled)):
    if y_pred[l]==y_test[l]:
        score=score+1
score=score*100/len(test_scaled)
print("Accuracy = " + "%.2f" % score + "%")
print(sum(y_pred))

#calculate no of flase negatives
score=0
pos=0
for i in range(len(test)):
    if y_pred[i]==0 and y_test[i]==1:
        score=score+1
    if y_test[i]==1:
        pos=pos+1

#score=score*100/len(test)
percent = score/len(test)

print("No of false negatives = %d" % score)
print("No of positives = %d" % pos)

```

    Accuracy = 94.19%
    1370.0
    No of false negatives = 35
    No of positives = 328
    Wall time: 6min 11s

```python

```


# Probability

In this notebook, we illustrate some basic concepts from probability theory using Python code.

```python

import os
import time
import numpy as np
np.set_printoptions(precision=3)
import glob
import matplotlib.pyplot as plt
import PIL
import imageio

import sklearn
import scipy.stats as stats
import scipy.optimize

import seaborn as sns;
sns.set(style="ticks", color_codes=True)

import pandas as pd
pd.set_option('precision', 2) # 2 decimal places
pd.set_option('display.max_rows', 20)
pd.set_option('display.max_columns', 30)
pd.set_option('display.width', 100) # wide windows

```

# Software libraries

There are several software libraries that implement standard probability distributions, and functions for manipulating them (e.g., sampling, fitting). We list some below.

* [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html)  We illustrate how to use this below.

* [Tensorflow probability (TFP)](https://www.tensorflow.org/api_docs/python/tf/distributions)
Similar API to scipy.stats.

* [Distrax](https://github.com/deepmind/distrax) JAX version of TFP.

* [Pytorch distributions library](https://pytorch.org/docs/stable/distributions.html). Similar to TFP.

* [NumPyro distributions library](https://numpyro.readthedocs.io/en/latest/distributions.html) has a similar interface to PyTorch distributions, but uses JAX as the backend.

In this notebook, we mostly focus on scipy.stats.

# Basics of Probability theory

## What is probability?

We will not go into mathematical detail, but focus on intuition.

*   Two main "schools of thought"
*   **Bayesian probability** = degree of belief
  * $p(heads=1)=0.5$ means you think the event that a particular coin will land heads is 50% likely.
* **Frequentist probability** = long run frequencies
  * $p(heads=1)=0.5$ means that the empirical fraction of times this event will occur across infinitely repeated trials is 50%
* In practice, the philosophy does not matter much, since both interpretations must satisfy the same basic axioms

## Random variables and their distributions

*   Let $X$ be a (discrete) **random variable** (RV) with $K$ possible values  $\mathcal{X}  = \{1,...,K\}$.
*   Let $X=x$ be the **event** that $X$ has value $x$, for some state $x \in \cal{X}$.
* We require $0 \leq p(X=x) \leq 1$
* We require
     $$\sum_{x \in  \cal{X}} p(X=x) = 1$$
* Let $p(X) = [p(X=1), …, p(X=K)]$ be the distribution or **probability mass function** (pmf) for RV $X$.
* We can generalize this to continuous random variables, which have an infinite number of possible states, using a **probability density function** (pdf) which satisfies
$$
    \int_{x \in \cal{X}} p(X=x) dx = 1
    $$

## Conjunction and disjunction of events
*   The probability of events $X=x$ AND $Y=y$ is denoted $p(X=x \land Y=y)$ or just $p(X=x,Y=y)$.

*   If two RVs are **independent**, then
$$p(X, Y) = p(X) * p(Y)$$.

*   The probability of event $X=x$ OR $Y=y$ is
$$p(X=x \lor Y=y) = p(X=x) + p(Y=y) - p(X=x \land Y=y)$$
*   For disjoint events (that cannot co-occur), this becomes
$$p(X=x \lor Y=y) = p(X=x) + p(Y=y)$$

## Conditional probability, sum rule, product rule, Bayes rule

* The **conditional probability** of Y=y given X=x is defined to be

 $$   p(Y=y|X=x) = \frac{p(X=x,Y=y)}{p(X=x)} $$

* Hence we derive the **product rule**
 $$
 \begin{align}
   p(X=x, Y=y) &= p(Y=y|X=x) * p(X=x)\\
              &= p(X=x|Y=y) * p(Y=y)
              \end{align}
  $$

* If $X$ and $Y$ are independent, then $p(Y|X)=p(Y)$ and $p(X|Y)=p(X)$, so
  $p(X,Y)=p(X) p(Y)$.

* The marginal probability of $X=x$ is given by the **sum rule**
   $$ p(X=x) = \sum_y p(X=x, Y=y)$$

* Hence we derive **Bayes' rule**
   $$
   \begin{align}
   p(Y=y|X=x) &= p(X=x,Y=y) /  p(X=x)\\
   &=\frac{p(X=x|Y=y) * p(Y=y)}
   {\sum_{y'} p(X=x|Y=y) * p(Y=y)}
   \end{align}
   $$

## Bayesian inference

Bayes rule is often used to compute a distribution over possible values of a **hidden variable** or **hypothesis** $h \in \cal{H}$ after observing some evidence $Y=y$. We can write this as follows:

$$
\begin{align}
p(H=h|Y=y) &= \frac{p(H=h) p(Y=y|H=h)}{p(Y=y)} \\
\text{posterior}(h|y) &= \frac{\text{prior}(h) * \text{likelihood}(y|h)}{\text{marginal-likelihood}(y)}
 \end{align}
 $$

* The **prior** encodes what we believe about the state before we see any data.
* The **likelihood** is the probability of observing the data given each possible hidden state.
* The **posterior** is our new belief state, after seeing the data.
* The **marginal likelihood** is a normalization constant, independent of the hidden state, so can usually be ignored.

Applying Bayes rule to infer a hidden quantity from one or more observations is called **Bayesian inference** or **posterior inference**. (It used to be called **inverse probability**, since it reasons backwards from effects to causes.)

## Example: Bayes rule for COVID diagnosis

Consider estimating if someone has COVID $H=1$ or not $H=0$ on the basis of a PCR test. The test can either return a positive result $Y=1$ or a negative result $Y=0$. The reliability of the test is given by the following observation model.

Using data from https://www.nytimes.com/2020/08/04/science/coronavirus-bayes-statistics-math.html, we set sensitivity to 87.5\% and the specificity to 97.5\%.

We also need to specify the prior probability $p(H=1)$; this is known as the prevalence. This varies over time and place, but let's pick $p(H=1)=0.1$ as a reasonable estimate.

If you test positive:

\begin{align}
p(H=1|Y=1)
 &= \frac{p(Y=1|H=1) p(H=1)}
{p(Y=1|H=1) p(H=1) + p(Y=1|H=0) p(H=0)}
= 0.795
\end{align}

If you test negative:
\begin{align}
p(H=1|Y=0)
 &= \frac{p(Y=0|H=1) p(H=1)}
{p(Y=0|H=1) p(H=1) + p(Y=0|H=0) p(H=0)}
=0.014
\end{align}

## Code to reproduce the above.

```python
def normalize(x):
  return x / np.sum(x)

def posterior_covid(observed, prevalence=None, sensitivity=None):
  # observed = 0 for negative test, 1 for positive test
  # hidden state = 0 if no-covid, 1 if have-covid
  if sensitivity is None:
    sensitivity = 0.875
  specificity = 0.975
  TPR = sensitivity;
  FNR = 1-TPR
  TNR = specificity
  FPR = 1-TNR
  # likelihood(hidden, obs)
  likelihood_fn = np.array([[TNR, FPR], [FNR, TPR]])
  # prior(hidden)
  if prevalence is None:
    prevalence = 0.1
  prior = np.array([1-prevalence, prevalence])
  likelihood = likelihood_fn[:, observed].T
  posterior = normalize(prior * likelihood)
  return posterior
```

For a prevalence of $p(H=1)=0.1$

```python
print(posterior_covid(1)[1]*100)
print(posterior_covid(0)[1]*100)

```

    79.54545454545453
    1.4044943820224722

For a prevalence of $p(H=1)=0.02$

```python

print(posterior_covid(1, 0.02)[1]*100)
print(posterior_covid(0, 0.02)[1]*100)
```

    41.66666666666664
    0.2609603340292276

# Univariate distributions

## Univariate Gaussian (normal)

```python
from scipy.stats import norm

rv = norm(0, 1) # standard normal

fig, ax = plt.subplots(1,2, figsize=(10,5))
X = np.linspace(-3, 3, 500)
ax[0].plot(X, rv.pdf(X))
ax[0].set_title("Gaussian pdf")
ax[1].plot(X, rv.cdf(X))
ax[1].set_title("Gaussian cdf")
plt.show()

plt.figure()
plt.plot(X, rv.pdf(X))
plt.title("Gaussian pdf")

plt.figure()
plt.plot(X, rv.cdf(X))
plt.title("Gaussian cdf")

plt.show()

```

```python
# Samples

np.random.seed(42)
mu = 1.1
sigma = 0.1
dist = norm(loc=mu, scale=sigma) # create "frozen" distribution
N = 10
x = dist.rvs(size=N) # draw N random samples
print(x.shape)
print(x)

np.random.seed(42)
x2 = norm(mu, sigma).rvs(size=N)
assert np.allclose(x, x2)

```

    (10,)
    [1.15  1.086 1.165 1.252 1.077 1.077 1.258 1.177 1.053 1.154]

```python
# pdf, cdf, inverse cdf

logprob = dist.logpdf(x) # evaluate log probability of each sample
print(logprob.shape)

p = dist.cdf(x)
x3 = dist.ppf(p) # inverse CDF
assert np.allclose(x, x3)
```

    (10,)

## Gamma distribution

```python
from scipy.stats import gamma

x = np.linspace(0, 7, 100)
b = 1
plt.figure()
for a in [1, 1.5, 2]:
    y = gamma.pdf(x, a, scale=1/b, loc=0)
    plt.plot(x, y)
plt.legend(['a=%.1f, b=1' % a for a in [1, 1.5, 2]])
plt.title('Gamma(a,b) distributions')
#save_fig('gammaDistb1.pdf')
plt.show()

```

## Zipf's law

In this section, we study the empirical word frequencies derived from H. G. Wells' book [The time machine](https://en.wikipedia.org/wiki/The_Time_Machine).
Our code is based on https://github.com/d2l-ai/d2l-en/blob/master/chapter_recurrent-neural-networks/lang-model.md

```python
import collections
import re
import urllib

url = 'https://raw.githubusercontent.com/probml/pyprobml/master/data/timemachine.txt'
bytes = urllib.request.urlopen(url).read()
string = bytes.decode('utf-8')
words = string.split()
print(words[:10])
words = [re.sub('[^A-Za-z]+', ' ', w.lower()) for w in words]
print(words[:10])

```

    ['The', 'Time', 'Machine,', 'by', 'H.', 'G.', 'Wells', '[1898]', 'I', 'The']
    ['the', 'time', 'machine ', 'by', 'h ', 'g ', 'wells', ' ', 'i', 'the']

```python
# Convert sequence of words into sequence of n-grams for different n

# Unigrams
wseq = words # [tk for st in raw_dataset for tk in st]
print('First 10 unigrams\n', wseq[:10])

# Bigrams
word_pairs = [pair for pair in zip(wseq[:-1], wseq[1:])]
print('First 10 bigrams\n', word_pairs[:10])

# Trigrams
word_triples = [triple for triple in zip(wseq[:-2], wseq[1:-1], wseq[2:])]
print('First 10 trigrams\n', word_triples[:10])
```

    First 10 unigrams
     ['the', 'time', 'machine ', 'by', 'h ', 'g ', 'wells', ' ', 'i', 'the']
    First 10 bigrams
     [('the', 'time'), ('time', 'machine '), ('machine ', 'by'), ('by', 'h '), ('h ', 'g '), ('g ', 'wells'), ('wells', ' '), (' ', 'i'), ('i', 'the'), ('the', 'time')]
    First 10 trigrams
     [('the', 'time', 'machine '), ('time', 'machine ', 'by'), ('machine ', 'by', 'h '), ('by', 'h ', 'g '), ('h ', 'g ', 'wells'), ('g ', 'wells', ' '), ('wells', ' ', 'i'), (' ', 'i', 'the'), ('i', 'the', 'time'), ('the', 'time', 'traveller')]

```python
# ngram statistics

counter = collections.Counter(wseq)
counter_pairs = collections.Counter(word_pairs)
counter_triples = collections.Counter(word_triples)

wordcounts = [count for _,count in counter.most_common()]
bigramcounts = [count for _,count in counter_pairs.most_common()]
triplecounts = [count for _,count in counter_triples.most_common()]

print('Most common unigrams\n', counter.most_common(10))
print('Most common bigrams\n', counter_pairs.most_common(10))
print('Most common trigrams\n', counter_triples.most_common(10))
```

    Most common unigrams
     [('the', 2232), ('and', 1197), ('i', 1176), ('of', 1147), ('a', 800), ('to', 671), ('was', 537), ('in', 535), ('my', 435), ('that', 404)]
    Most common bigrams
     [(('of', 'the'), 309), (('in', 'the'), 168), (('i', 'had'), 127), (('and', 'the'), 109), (('i', 'was'), 107), (('the', 'time'), 93), (('it', 'was'), 88), (('to', 'the'), 85), (('of', 'a'), 71), (('as', 'i'), 70)]
    Most common trigrams
     [(('the', 'time', 'traveller'), 32), (('the', 'time', 'traveller '), 20), (('the', 'time', 'machine '), 19), (('the', 'medical', 'man '), 17), (('it', 'seemed', 'to'), 15), (('i', 'did', 'not'), 14), (('i', 'began', 'to'), 13), (('it', 'was', 'a'), 13), (('said', 'the', 'time'), 12), (('out', 'of', 'the'), 12)]

```python
# Word frequency is linear on log-log scale
plt.figure()
plt.loglog(wordcounts, label='word counts');
plt.ylabel('log frequency')
plt.xlabel('log rank')

# Prediction from Zipf's law, using manually chosen parameters.
# We omit the first 'skip' words, which don't fit the prediction well.
skip = 10.0
x = np.arange(skip, len(wordcounts))
N = np.sum(wordcounts)
kappa = 0.1
a = -1
y = kappa*np.power(x, a) * N # predicted frequency for word with rank x
plt.loglog(x, y, label='linear prediction')

plt.legend()
plt.show()

```

```python
# The number of unique n-grams is smaller for larger n.
# But n-gram statistics also exhibit a power law.

plt.figure()
plt.loglog(wordcounts, label='word counts');
plt.loglog(bigramcounts, label='bigram counts');
plt.loglog(triplecounts, label='triple counts');
plt.legend();
plt.show()

```

```python

```

# Multivariate Gaussian (normal)

```python
from scipy.stats import multivariate_normal as mvn

D = 5
np.random.seed(42)
mu = np.random.randn(D)
A = np.random.randn(D,D)
Sigma = np.dot(A, A.T)

dist = mvn(mu, Sigma)
X = dist.rvs(size=10)
print(X.shape)
```

    (10, 5)

```python
from mpl_toolkits import mplot3d
from mpl_toolkits.mplot3d import Axes3D

names = ["Full", "Diag", "Spherical"]

mu = [0, 0]
Covs = {'Full': [[2, 1.8], [1.8, 2]],
        'Diag': [[1, 0], [0, 3]],
        'Spherical': [[1, 0], [0, 1]]}

N = 100
points = np.linspace(-5, 5, N)
X, Y = np.meshgrid(points, points)
xs = X.reshape(-1)
ys = Y.reshape(-1)
grid = np.vstack([xs, ys]).T # N^2 * 2

fig = plt.figure(figsize=(10,7))
fig.subplots_adjust(hspace=0.5, wspace=0.1)
fig_counter = 1
for i in range(len(Covs)):
    name = names[i]
    Sigma = Covs[name]
    ps = mvn(mu, Sigma).pdf(grid)
    P = ps.reshape((N,N))

    ax = fig.add_subplot(3, 2, fig_counter)
    ax.contour(X, Y, P)
    ax.axis('equal') # make circles look circular
    ax.set_title(name)
    fig_counter = fig_counter + 1

    ax = fig.add_subplot(3, 2, fig_counter, projection='3d')
    ax.plot_surface(X, Y, P, rstride=2, cstride=2)
    ax.set_title(name)
    fig_counter = fig_counter + 1
plt.show()
```

Illustrate correlation coefficient.

Code is from [Bayesian Analysis with Python, ch. 3](https://github.com/aloctavodia/BAP/blob/master/code/Chp3/03_Modeling%20with%20Linear%20Regressions.ipynb)

```python

sigma_x1 = 1
sigmas_x2 = [1, 2]
rhos = [-0.90, -0.5, 0, 0.5, 0.90]

k, l = np.mgrid[-5:5:.1, -5:5:.1]
pos = np.empty(k.shape + (2,))
pos[:, :, 0] = k
pos[:, :, 1] = l

f, ax = plt.subplots(len(sigmas_x2), len(rhos),
                     sharex=True, sharey=True, figsize=(12, 6),
                     constrained_layout=True)
for i in range(2):
    for j in range(5):
        sigma_x2 = sigmas_x2[i]
        rho = rhos[j]
        cov = [[sigma_x1**2, sigma_x1*sigma_x2*rho],
               [sigma_x1*sigma_x2*rho, sigma_x2**2]]
        rv = stats.multivariate_normal([0, 0], cov)
        ax[i, j].contour(k, l, rv.pdf(pos))
        ax[i, j].set_xlim(-8, 8)
        ax[i, j].set_ylim(-8, 8)
        ax[i, j].set_yticks([-5, 0, 5])
        ax[i, j].plot(0, 0,
                      label=f'$\\sigma_{{x2}}$ = {sigma_x2:3.2f}\n$\\rho$ = {rho:3.2f}', alpha=0)
        ax[i, j].legend()
f.text(0.5, -0.05, 'x_1', ha='center', fontsize=18)
f.text(-0.05, 0.5, 'x_2', va='center', fontsize=18, rotation=0)
```


# Algebra BUG HUNT!

```python
# create a list
lst = [1;3;4;1;6]

print( lst )
```

```python
# add all the numbers together
print( np.add(lst) )
```

```python
# plot the cumulative sum of a list of numbers
l = np.arange(-4,10)

plt.plot(l,'rs-')
plt.plot(np.sum(1),'bo-')
plt.legend({'list','cumsum'})
plt.show()
```

```python
### the equation:
# 4 - 2x + 5x^3

# define the coefficients
coefs = [4,-2,5]

# solve
roots = np.roots(coefs)

# and display
from sympy.abc import x

p = sym.Poly(coefs,x)

display(Math(sym.latex(p)))
```

```python
def quadeq(a,b,c):

    # compute
    out[0] = ( -b - sp.sqrt(b**2 - 4*a*c) ) / (2*a)
    out[1] = ( -b + sp.sqrt(b**2 - 4*a*c) ) / (2*a)

    # output
    return out

# test the function
print(quadeq(1,4,2))
```

```python
# create a complex number

real_part = 4
imag_part = -6

cn = np.complex(imag_part,real_part)

plt.plot(np.real(cn),np.imag(cn),'ro')
plt.grid('on')
plt.axis('square')
plt.axis([-10,10,-10,10])
plt.show()
```

```python
# symbolic complex number
a,b = sym.symbols('a,b',real=True)

z = a + b*sym.1j
display(Math('z\\times z^* = %s' %sym.latex(sym.expand(z*sym.conjugate(z)))))
```

```python
# define the phase angles
x = np.linspace(0,2*np.pi,100)

# generate the plot
plt.plot(np.cos(x),np.sin(x),'k')

# draw one vector from the origin
phs = 1*np.pi/4
plt.plot([0,np.cos(phs)],[0,np.sin(phs)],'r-')
plt.plot(np.cos(phs),np.sin(phs),'ro')

# draw axis lines
plt.plot([-1.3,1.3],[0,0],'--',color=[.8,.8,.8])
plt.plot([0,0],[-1.3,1.3],'--',color=[.8,.8,.8])

# make it look nicer
plt.axis('square')
plt.axis([-1.3,1.3,-1.3,1.3])
plt.xlabel('sine(x)')
plt.ylabel('cosine(x)')
plt.plot()

plt.show()
```

```python
a =   2 # lower bound
b = 100 # upper bound
n =   3 # number of steps

lo = np.logspace(np.log10(a),np.log10(b))
li = np.linspace(a,b)

plt.plot(li,lo,'s-',label='log')
plt.plot(li,li,'o-',label='linear')

plt.legend(['linear','log'])
plt.axis('square')
plt.show()
```

```python
## Goal is to plot the point on the function closest to f(x)=.5

# x range
x = np.linspace(-6,10,20)

# the function f(x)
fx = 1/(1+np.exp(x))

# function maximum
fmaxidx = np.argmin(abs(fx-.5))

# draw the function
plt.plot(x,fx,'bo-')
plt.plot(fmaxidx,fx[fmaxidx],'rs')

plt.show()
```

```python
## Goal is to find the local minima by adding one character

from scipy.signal import find_peaks

# x range
x = np.linspace(0,12*np.pi,200)

# the function f(x)
fx = -( np.cos(x) + x**(1/2) )

# find peaks
peeks = find_peaks(fx)

# draw the function
plt.plot(x,fx)
plt.plot(x[peeks[0]],fx[peeks[0]],'o')
plt.show()

```


# V2: SCF optimization with VAMPyR

## V2.1: Hydrogen atom

In order to solve the one-electron Schr\"{o}dinger equation in MWs we reformulate them in an integral form [1].

\begin{equation}
 \phi = -2\hat{G}_{\mu}\hat{V}\phi
\end{equation}

Where $\hat{V}$ is the potential acting on the system, $\phi$ is the wavefunction, $\hat{G}$ is the Helmholtz integral operator, where its kernel is defined as $G_\mu(r - r') = \frac{\exp(-\mu |r - r'|)}{4\pi |r - r'|}$
and $\mu$ is a parameter defined above through the energy.

The Helmholtz operator is already implemented in vampyr, therefore the only things you need are the integral KS equation and the definition of $\mu$

\begin{equation}
     \mu = \sqrt{-2E}
\end{equation}

The way you initialize the Helmholtz operator is as follows
```
H = vp.HelmholtzOperator( mra, exp=mu, prec=eps )
```
where `mu` is the $\mu$ is the parameter defined above, mra you have seen before, and `eps` is the desired threshold precision. This operator is applied the same way you applied the vp.ScalingProjector earlier.

In this exercise you will be solving the KS equation iteratively for a simple system, the Hydrogen atom. This means that you only have the nuclear potential to take into account for the potential term in the KS equation.

$$    V_{nuc}(\mathbf{r}) = -\frac{1}{|\mathbf{r}|}$$

We will also be working with a single orbital, of which the initial guess is

$$    \phi_0(\mathbf{r}) = e^{-|\mathbf{r}|^2} $$
where
$$    |\mathbf{r}| = \sqrt{x^2 + y^2 + z^2}$$

The orbital update is defined as follows
\begin{align}
    \Delta\tilde{\phi}^n &= -2\hat{G}[V_{nuc}\phi^n] - \phi^n \\
    \Delta\tilde{\phi}^n &= \tilde{\phi}^{n+1} - \phi^n
\end{align}
where we use \~ to denote a function that is **not** normalized, and $n$ is the iteration index.

#### Implementation exercise:

1. Make a nuclear potential as a python function `f_nuc(r)`
2. Make an initial guess for the orbital as a python function `f_phi(r)` (hint use `np.exp` to get an exponetial function)
3. Create a Helmholtz operator $G_\mu$ with $\mu$ as shown above, use the exact value of $E = -0.5 a.u.$ for a hydrogen atom
4. Project both nuclear potential ($V$) and orbital ($\phi_n$) to the MW basis using a `vp.ScalingProjector` with precision $\epsilon=1.0e-3$
5. Compute new orbital through application of the Helmholtz operator
6. Compute the size of the orbital update $||\tilde{\phi}^{n+1} - \phi^n||$
7. Normalize the orbital $\phi^{n+1} = \tilde{\phi}^{n+1}/||\tilde{\phi}^{n+1}||$
8. Update orbital $\phi^{n+1} \rightarrow \phi^{n}$ for next iteration
9. Repeat steps 5-8 until your wavefunction has converged

The convergence criterion is the norm of $\Delta \phi^n$, but you should start by looping a set amount of times before trying the threshold.

```python
from vampyr import vampyr3d as vp
import numpy as np
import matplotlib.pyplot as plt

r_x = np.linspace(-0.99, 0.99, 1000) # create an evenly spaced set of points between -0.99 and 0.99
r_y = np.zeros(1000)
r_z = np.zeros(1000)
r = [r_x, r_y, r_z]

# Analytic nuclear potential
def f_nuc(r):
    # TODO: implement the nuclear potential
    return

# Analytic guess for solution
def f_phi(r):
    # TODO: implement the initial guess for the orbital
    return

# Prepare Helmholtz operator
E = -0.5
mu = np.sqrt(-2*E)
G_mu =   # TODO: Construct BSH operator from mu)

V =      # TODO: Project nuclear potential V from f_nuc
phi_n =  # TODO: Project starting guess phi_n from f_phi
phi_n.normalize()

# Optimization loop
thrs = 1.0e-3
update = 1.0
i = 0
while (i < 3): # switch to (update > thrs) later
    # TODO:
    # Compute product of potential V and wavefunction phi_n
    # Apply Helmholtz operator to obtain phi_np1
    # Compute norm = ||phi^{n+1}||
    # Compute update = ||phi^{n+1} - phi^{n}||

    # this will plot the wavefunction at each iteration
    phi_n_plt = [phi_n([x, 0.0, 0.0]) for x in r_x]
    plt.plot(r_x, phi_n_plt)

    # this will print some info, you need to compute in the loop:
    print("iteration: {}    Norm: {}   Update: {}".format(i, norm, update))
    i += 1

plt.show()
```

## V2.2 Extension to Helium

A few things change when you go from Hydrogen to Helium:
1. The energy is no longer known exactly, and thus will have to be computed from the wavefunction
2. The Helmholtz operator which depends on the energy through $\mu = \sqrt{-2E}$ needs to be updated in every iteration
3. The potential operator $V$ depends on the wavefunction and must be updated in every iteration

In this example we will use the Hartree-Fock model, which for a single-orbital system like Helium, reduces to the following potential operator:
\begin{align}
  \hat{V} &= \hat{V}_{nuc} + 2\hat{J} - \hat{K}\\
          &= \hat{V}_{nuc} + \hat{J}
\end{align}
since $\hat{K} = \hat{J}$ for a doubly occupied single orbital.

The Coulomb potential $\hat{J}$ can be computed by application of the Poisson operator $P$:
\begin{equation}
  \hat{J}(r) = P\left[4\pi\rho\right]
\end{equation}
Where $\rho$ is the square of the orbital
\begin{equation}
  \rho = \phi*\phi
\end{equation}

#### Pen and paper exercise:

Use the fact that
\begin{equation}
  \tilde{\phi}^{n+1} = -\Big[\hat{T} - E^n\Big]^{-1} V^n\phi^n \end{equation}
to show that
\begin{equation}
  E^{n+1} = \frac{\langle\tilde{\phi}^{n+1}|\hat{T} +
            \hat{V}^{n+1}|\tilde{\phi}^{n+1}\rangle}
            {||\tilde{\phi}^{n+1}||^2}
\end{equation}
can be written as a pure update $dE^n$ involving only the potentials $\hat{V}^{n+1}$, $\hat{V}^n$ as well as the orbitals $\tilde{\phi}^{n+1}$ and $\phi^n$
\begin{equation}
  E^{n+1} = E^{n} + dE^n
\end{equation}

#### Implementation exercise:
1. Make a nuclear potential function `f_nuc(r)` for the Helium atom
2. Make an initial guess for the orbital as a python function `f_phi(r)` (hint use `np.exp` to get an exponetial function)
3. Project both nuclear potential ($V$) and orbital ($\phi_n$) to the MW basis using a `vp.ScalingProjector` with precision $\epsilon=1.0e-3$
4. Create a Helmholtz operator $G^n$ with $\mu^n$ using the current energy $E^n$
5. Compute total potential $\hat{V^n} = \hat{V}_{nuc} + \hat{J^n}$, where the Coulomb potential is computed using the `vp.PoissonOperator` on the current squared orbital $\rho^n = ||\phi^n||^2$
6. Compute new orbital through application of the Helmholtz operator on $\phi^{n+1} = -2\hat{G}^n\hat{V}^n\phi^n$
7. Compute the size of the orbital update $||\tilde{\phi}^{n+1} - \phi^n||$
8. Normalize the orbital $\phi^{n+1} = \tilde{\phi}^{n+1}/||\tilde{\phi}^{n+1}||$
9. Update orbital $\phi^{n+1} \rightarrow \phi^{n}$ for next iteration
10. Repeat steps 4-9 until your wavefunction has converged

The convergence criterion is the norm of $\Delta \phi^n$, but you should start by looping a set amount of times before trying the threshold.

```python
from vampyr import vampyr3d as vp
import numpy as np
import matplotlib.pyplot as plt

r_x = np.linspace(-0.99, 0.99, 1000) # create an evenly spaced set of points between -0.99 and 0.99
r_y = np.zeros(1000)
r_z = np.zeros(1000)
r = [r_x, r_y, r_z]

# Analytic nuclear potential Helium
def f_nuc(r):
    #implement the nuclear potential
    return

# Analytic guess for solution (same as for Hydrogen)
def f_phi(r):
    # implement the initial guess for the orbital
    return

# TODO:
# Project nuclear potential V_nuc from f_nuc
# Project starting guess phi_n from f_phi
# Set a starting guess E_n for the energy

# Optimization loop
thrs = 1.0e-3
update = 1.0
i = 0
while (i < 3): # switch to (update > thrs) later

    # Prepare Helmholtz operator from current energy
    mu_n = np.sqrt(-2*E_n)
    G_n = # TODO: Construct BSH operator from mu_n)

    # TODO:
    # Compute rho
    # Initialize vp.PoissonOperator and compute J
    # Compute total potential V = V_nuc + J
    # Iterate Helmholtz operator to get new orbital phi^{n+1}

    dE_n = # TODO: insert energy expression from above

    # Prepare for next iteration
    E_n += dE_n
    phi_n += dPhi_n

    # This will plot the wavefunction at each iteration
    phi_n_plt = [phi_n([x, 0.0, 0.0]) for x in r_x]
    plt.plot(r_x, phi_n_plt)

    # this will print some info, you need to compute in the loop:
    # norm = ||phi^{n+1}||
    # update = ||phi^{n+1} - phi^{n}||
    print("iteration: {}  Energy: {}    Norm: {}   Update: {}".format(i, E_n, norm, update))
    i += 1

plt.show()
```

You should expect the orbital energy to converge towards
$E_n \approx -0.918$.

#### Bonus exercise:
The total energy can be computed after convergence as
$E_{tot} = 2E_n - \langle\rho|J\rangle$, should be around $E_{tot} \approx -2.86$.

##  Sources

[1] Stig Rune Jensen, Santanu Saha, José A. Flores-Livas, William Huhn, Volker Blum, Stefan Goedecker, and Luca Frediani The Elephant in the Room of Density Functional Theory Calculations. The Journal of Physical Chemistry Letters 2017 8 (7), 1449-1457
DOI: 10.1021/acs.jpclett.7b00255

```python

```


* Observer and feedback controller design for rear wheel speed control.

* Design notes:
  - US Digital optical encoder has 50 CPR of encoder shaft.
  - 100:25 wheel to encoder gearing implies 200 CPR of wheel.
  - Detecting rising and falling edges of A or B channels results in 400 CPR
    of wheel (half quadrature).
  - Detecting rising and falling edges of A and B channels results in 800 CPR
    (full quadrature).
  - Wheel angle measurement uses full quadrature.  One timer count represents
    2*pi/800 radians
  - Wheel speed measurement uses half quadrature.  One cycle between a rising
    and falling edge represents 2*pi/400 radians.
  - Encoder rising and falling edges occur asynchronously from main control
    loop.
  - Absolute position of wheel is generally not of interest.
  - Speed of wheel is of primary interest because bicycle dynamics are dependent upon speed.

```
from sympy import symbols, Poly
import numpy as np

c, J, kT, T, s, w, x, z, I = symbols('c J kT T s w x z I')
#bilinear_transform = {s : 2/T*(z-1)/(z+1)}
forward_difference_transform = {s : (z-1)/T}

# Continuous time transfer function from I(s) to w(s) of the following plant
# model:
#   dw/dt = -c/J*w + kT/J*I
G_s = kT/J/(s + c/J)

G_z_num, G_z_den = G_s.subs(forward_difference_transform).as_numer_denom()
G_z_den = Poly(G_z_den, z)

G_z_num = Poly(G_z_num / G_z_den.LC(), z) # divide by leading coefficient of den
G_z_den = G_z_den.monic()                 # make denominator monic
assert(G_z_den.coeffs()[0] == 1)
print(G_z_num)
print(G_z_den)
print(G_z_num / G_z_den)

A = - G_z_den.coeffs()[1]
B = G_z_num.coeffs()[0]
C = 1

```

    Poly(T*kT/J, z, domain='ZZ(J,T,kT)')
    Poly(z + (-J + T*c)/J, z, domain='ZZ(c,J,T)')
    T*kT/(J*(z + (-J + T*c)/J))

```
for n, m in [('A', A), ('B', B), ('C', C)]: print(n + " = " + str(m))

```

    A = -(-J + T*c)/J
    B = T*kT/J
    C = 1

```

```


# Modelamiento Bayesiano

En el enfoque Bayesiano:
- La inferencia se realiza calculado la distribución **a posteriori** de los parámetros
- Se modela la incerteza de los datos y parámetros con una **distribución de probabilidad conjunta**
- $\theta$ es un vector aleatorio cuya distribución de probabilidad se denomina **a priori**

El teorema de Bayes y la ley de probabilidades totales nos permiten escribir:

$$
p(\theta| \cal{D}) = \frac{p(\cal{D}, \theta)}{p(\cal{D})}= \frac{p(\cal{D}|\theta) p(\theta)}{\int p(\cal{D}|\theta) p(\theta) d\theta} \propto p(\cal{D}|\theta) p(\theta),
$$

En el enfoque Bayesiano buscamos calcular la distribución a posteriori, la que se construye a partir de la función de verosimilitud, la distribución a priori y la evidencia, conocida también como la verosimilitud marginal de los datos.

## El procedimiento de inferencia Bayesiana:

1. Definir la función de verosimilitud de los datos
2. Escoger una distribución a priori para los parámetros
3. Construir la distribución conjunta de datos y parámetros
4. Determinar la distribución a posteriori usando el Teorema de Bayes
5. Calcular estimadores puntuales:

**Moda a posteriori (MAP, maximum a posteriori)**
$$
\hat \theta = \text{arg} \max_\theta p(\theta|\cal{D}) =  \text{arg} \max_\theta p(\cal{D}| \theta) p(\theta),
$$

para esta maximización no se requiere la evidencia pues esta no depende de $\theta$. Además aplicando logaritmo podemos separar la verosimilitud de la distribución a priori:

$$
\hat \theta = \text{arg} \max_\theta (\log p(\cal{D}| \theta) + \log p(\theta)),
$$

**Esperanza a posteriori**
$$\mathbb{E}(\theta \mid \cal{D}) = \int \theta p(\theta \mid \cal{D}) d\theta$$

**Varianza a posteriori**
$$\mathbb{V}(\theta \mid \cal{D}) = \int (\theta- \mathbb{E}(\theta \mid \cal{D}))^2 p(\theta \mid \cal{D}) d\theta$$

### Algunas dificultades con MAP:
- no es necesariamente representativa de la distribución a posteriori
- no es invariante a reparametrizaciones de los parámetros

```R
library("IRdisplay")
display_png(file="graficoMAP0.png")
display_png(file="graficoMAP.png")
```

### Intervalos de credibilidad:
- Intervalos centrales

$$C_{\alpha}({\cal D}) = (l,u) : P(l \leq \theta \leq u \mid {\cal D}) = 1-\alpha$$

- Regiones de mas altas densidades a posteriori (HPD):

Sea $p*$ un umbral para la distribución a posteriori que cumple que:
$$1-\alpha = \int_{\theta: p(\theta\mid {\cal D})>p*} p(\theta \mid {\cal D}) d\theta$$

entonces se define:
$$C_{\alpha}({\cal D}) = \{ \theta : p(\theta\mid {\cal D})\geq p*\}$$

```R
display_png(file="graficoCentral.png")
```

**Distribución Predictiva a posteriori**
$$p(x \mid \cal{D}) = \int p(x \mid \theta) p(\theta \mid \cal{D}) d\theta$$

Nos permite predecir la probabilidad de ocurrencia de un nuevo dato $x$

## Distribuciones a priori

Distribuciones naturales conjugadas: Una elección posible para las distribuciones a priori son aquellas que se conjugan con las funciones  de verosimilitud, de manera que la distribución a posteriori es del mismo tipo que la distribución a priori pero con nuevos parámetros. Veamos algunos ejemplos.

**i) El modelo Beta-binomial**

Consideremos el caso en que

$$ X_i \sim i.i.d. \, Binomial(K,\theta) \qquad X_i \in \{0,...,K\}, i=1,\cdots,n$$

$$ P(X_i = k\mid \theta) = {{K}\choose{k}} \theta^k (1-\theta)^{(K-k)} \qquad k = 1,...,K$$

Y por otra parte
$$ \theta \sim Beta(a,b) \qquad \theta \in [0,1], a,b>0$$

$$f_{\theta}(\theta) = \frac{\Gamma(a+b)}{ \Gamma(a) \Gamma(b)} \theta^{a-1} (1-\theta)^{b-1} \qquad \theta \in [0,1]$$

```R
library(ggplot2)
options(repr.plot.width = 8, repr.plot.height = 6)
# Grafico densidad Beta con distintos parámetros
vec <- seq(0,1,by=0.01)
fvec <- data.frame(cbind(vec,dbeta(vec,0.1,0.1),dbeta(vec,1,1),dbeta(vec,2,3),dbeta(vec,8,4)))

h<- ggplot(fvec, aes(x=vec))
h <- h +coord_cartesian(ylim=c(0,3))
h <- h + geom_line(y=fvec$V5,color="green", linetype=1,size=0.8)
h <- h + geom_line(y=fvec$V4,color="red",size=0.8, linetype=2)
h <- h + geom_line(y=fvec$V3,color="blue",size=0.8, linetype=3)
h <- h + geom_line(y=fvec$V2,color="violet", size=0.8,linetype=4)
h
```

**Algunas Propiedades de la Distribución Beta**

i) $moda(\theta)= \underset{\theta}{\operatorname{argmax}} f_{\theta}(\theta)$

$$\begin{align}
\frac{\partial f_{\theta}(\theta)}{\partial \theta}=0 \qquad &\implies \qquad
(a-1)\theta^{a-2}(1-\theta)^{b-1} -  (b-1)\theta^{a-1}(1-\theta)^{b-2} =0 \nonumber\\
&\implies \qquad \theta^{a-2}(1-\theta)^{b-2}((a-1)(1-\theta) - (b-1)\theta) =0 \nonumber\\
&\implies \qquad \theta(a-1 + b-1) =a-1 \nonumber\\
&\implies \qquad moda(\theta) = \frac{a-1}{a+b-2} \nonumber\\
\end{align}$$

ii) $\mathbb{E}(\theta) = \int_0^1 \theta f_{\theta}(\theta) d\theta$
$$\begin{align}
\mathbb{E}(\theta) & = \frac{\Gamma(a+b)}{ \Gamma(a) \Gamma(b)} \int_0^1 \theta \theta^{a-1} (1-\theta)^{b-1}d\theta \nonumber\\
& =  \frac{a}{a+b}
\nonumber\\
\end{align}$$

iii) $\mathbb{V}(\theta) = \int_0^1 (\theta- \mathbb{E}(\theta)^2 f_{\theta}(\theta) d\theta$
$$\begin{align}
\mathbb{V}(\theta) & = \frac{\Gamma(a+b)}{ \Gamma(a) \Gamma(b)} \int_0^1 (\theta- \mathbb{E}(\theta)^2 \theta^{a-1} (1-\theta)^{b-1}d\theta \nonumber\\
& =  \frac{ab}{(a+b)^2(a+b+1)}
\nonumber\\
\end{align}$$

**Nota:**
$$\Gamma(z) = \int_0^{\infty} t^{z-1} e^{-t} dt$$
y cumple:
$$\Gamma(z+1) = z \Gamma(z), \qquad \Gamma(n) = (n-1)! , n\in N$$

**Cálculo de la distribución a posteriori**

$$
\begin{align}
f_{\theta}(\theta \mid \{X_i\})
&\propto \prod_{i=1}^n P(X_i=k_i\mid \theta)f_{\theta}(\theta) \nonumber \\
&\propto \prod_{i=1}^n \theta^{k_i} (1-\theta)^{(K-k_i)} \theta^{a-1} (1-\theta)^{b-1} \nonumber \\
&\nonumber\\
&= \theta^{N_1} (1-\theta)^{N_0} \theta^{a-1} (1-\theta)^{b-1} \nonumber \\
& \nonumber\\
&= \theta^{N_1+a-1} (1-\theta)^{N_0+b-1}
\end{align}
$$
donde
$$N_1 = \sum_{i=1}^n k_i \qquad y \qquad
N_0 = \sum_{i=1}^n (K-k_i)$$
Es decir
$$ \theta \mid \{X_i\}_{i=1,..,n} \sim Beta(N_1+a,N_0+b)$$

**Moda a posteriori (MAP):**
$$\theta_{MAP}= \underset{\theta}{\operatorname{argmax}} f_{\theta}(\theta \mid \{x_i\})= \frac{a+N_1-1}{a+b+N-2}$$

**Media a posteriori:**

$$\mathbb{E}(\theta \mid \{x_i\}_{i=1,...,n})  = \frac{a+N_1}{a+b+N}$$

**Varianza a posteriori**
$$\mathbb{V}(\theta \mid \{x_i\}_{i=1,...,n})  = \frac{(a+N_1)(b+N_2)}{(a+b+N)^2(a+b+N+1)}$$

```R

display_png(file="graficosBeta.png")
```

**Ejercicio 1:** Calcule la distribución a posteriori para el caso del modelo Beta-Bernoulli.

**Ejercicio 2:** Calcule las siguientes probabilidades predictivas a posteriori:

i) $P(x=1 \mid \cal{D})$ considerando que hay un sólo ensayo ( o sea K=1).

ii)$P(x \mid \cal{D}, M)$ considerando que hay M ensayos.

**i) El modelo Dirichlet-multinomial**

Consideremos el caso en que

$$ {\bf{X}}=(x_1,....x_K) \sim Multinomial(n,{\bf{\theta}}) \qquad x_i \in \{0,...,n\}, n=\sum_{k=1}^K x_k , \qquad {\bf{\theta}}=(\theta_1,...,\theta_K)  $$

$$ P(X = (x_1,...,x_k) \mid {\bf{\theta} }) = {{n}\choose{x_1,...,x_k}}\prod_{k=1}^K\theta_k^{x_k} = \frac{n!}{x_1!...x_k!}\prod_{k=1}^K\theta_k^{x_k}$$
De manera que la función de verosimilitud para n observaciones queda:
$$ P(\cal{D} \mid \bf{\theta}) \propto \prod_{k=1}^K\theta_k^{N_k}$$
donde
$$N_k = \sum_{i=1}^n (X_i)_k$$

Por otra parte, sea
$$ \theta = (\theta_1,...\theta_K) \sim Dirichlet(\alpha_1,...,\alpha_K) \qquad \theta_i \in [0,1], \sum_{k=1}^K \theta_i= 1, \alpha_i>0$$

$$f_{\theta}(\theta) = \frac{1}{B(\alpha)} \prod_{k=1}^K \theta_k^{\alpha_k-1} \qquad B(\alpha) = \frac{\prod_{k=1}^K \Gamma(\alpha_k)}{\Gamma(\alpha_1+...+\alpha_K)}$$

En este caso, resulta que la distribución a posteriori es también Dirichlet:
$$
\begin{align}
f_{\theta}(\theta \mid \cal{D}) & \propto P(\cal{D} \mid \theta)f_{\theta}(\theta) \nonumber \\
& \propto \prod_{k=1}^K \theta_k^{N_k}\theta_k^{\alpha_k-1} = \prod_{k=1}^K \theta_k^{N_k+\alpha_k-1} \nonumber
\end{align}
$$
Es decir

$$(\theta \mid {\cal{D}}) \sim Dirichlet(\alpha_1+N_1,...,\alpha_K+N_k)$$

### Ejemplo: diferencia de proporciones
Supongamos que estamos interesados en comprar en una plataforma online, y hay dos vendedores que entregan el producto al mismo precio. El vendedor 1 tiene 90 evaluaciones positivas y 10 negativas. El vendendor 2 tiene 2 evaluaciones positivas y 0 negativas. ¿Cómo decidir cual comprar?
En el enfoque bayesiano, el análisis es como sigue:

Sean $\theta_1$ y $\theta_2$ las confiabilidades reales de los dos vendedores. Como no tenemos mas información, vamos a considerar a priori uniformes: $\theta_i \sim Beta(1,1)$ y entonces podemos usar el modelo beta-binomial. De manera que
$$\theta_i \mid {\cal D} \sim Beta(1+y_i, 1 + N_i - y_i), \qquad i=1,2$$
En este caso $$N_1 = 100, \,y_1 = 90,\qquad N_2 = 2,\, y_2 =2$$

Entonces podemos calcular, para $\delta = \theta_1 - \theta_2$:

$$P(\delta > 0 \mid {\cal D}) = \int_0^1 \int_0^1 \mathbb{1}_{\{\theta_1 - \theta_2>0\} }Beta(1+y_1, 1 + N_1 - y_1)$$
$$\qquad \qquad \qquad \qquad Beta(1+y_2, 1 + N_2 - y_2)d\theta_1 d\theta_2$$

Utilizando integración numérica se puede obtener que $P(\delta > 0 \mid {\cal D})= 0.71$ y un intervalo central de la figura.

```R
display_png(file="proporciones.png")
```

## Enfoque Bayesiano para la Selección de modelos
Si disponemos de un conjunto de modelos entre los cuales elegir, por ejemplo una familia de distribuciones de probabilidad paramétricas de complejidad diferente, estamos en presencia del problema de selección de modelos.

```R
display_png(file="overfitting.png")
```

En el enfoque frecuentista, una manera de abordar esto es usar validación cruzada (cross-validation), para definir los errores para cada modelo y luego seleccionar el de mínimo error.

En el enfoque Bayesiano podemos hacer uso de la distribución predictiva de los datos o evidencia, por cada modelo $m$ y escoger el de mayor probabilidad dados los datos $\cal{D}$
$$p(m \mid \cal{D}) = \frac{p(\cal{D} \mid m) p(m) }{\sum_{m \in \cal{M}}p(\cal{D} \mid m) p(m)}$$

y entonces calculamos el modelo MAP:
$$\hat{m} = \underset{m \in \cal{M}}{\operatorname{argmax}}p(m \mid \cal{D})$$

Si consideramos una distribución  a priori no informativa sobre los modelos (distribución uniforme), es decir $p(m) \propto 1$, tenemos que:
$$p(m \mid \cal{D}) \propto p(\cal{D} \mid m) = \int p(\cal{D} \mid \theta) p(\theta \mid m) d\theta$$

llamada verosimilitud marginal o evidencia del modelo $m$.

**La navaja de Occam:** (Bayesian Occam's razor): el hecho de que la verosimilitud marginal considere la integración respecto de los parámetros impide que se maximice simplemente por aumentar el número de parámetros, con lo cual se previene el sobreajuste en la elección del modelo.

```R
display_png(file="seleccion1.png")
```

```R
display_png(file="seleccion2.png")
```

El cálculo de la verosimilitud marginal o evidencia es computacionalmente costoso en general y requiere utilizar técnicas del tipo Monte Carlo, salvo cuando tratamos con modelos de distribuciones conjugadas.

Veamos el **caso del modelo Beta-Binomial:**

En esta caso $$\cal{M} = \{ m_{a,b} \mid \theta \sim Beta(a,b), a,b>0 \}$$

Para el cálculo de $p(\cal{D} \mid m)$, recordemos que en este caso la distribución a posteriori se distribuye $Beta(a+N_1,b+N_0)$, entonces considerando Bayes tenemos:

$$
\begin{align}
p(\cal{D} \mid m) &= \frac{p(\cal{D} \mid \theta, m) p(\theta \mid m)}{p(\theta \mid \cal{D},m)} \nonumber \\
&\\
&= \frac{\left({{N}\choose{N_1}}\theta^{N_1-1}(1-\theta)^{N_0-1}\right)\left( \frac{1}{B(a,b)}\theta^{a-1}(1-\theta)^{b-1}\right)}
{\left(\frac{1}{B(a+N_1,b+N_0)}\theta^{a+N_1-1}(1-\theta)^{b+N_0-1}\right)}\\
&\\
&= {{N}\choose{N_1}} \frac{B(a+N_1,b+N_0)}{B(a,b)}
\end{align}
$$

#### Aproximación BIC del logaritmo de la verosimilitud marginal
Considerando que el cálculo de la verosimilitud marginal es en general costoso, Schwartz(1978) propone el Criterio de Información Bayesiano (BIC), como una aproximación del logaritmo de la verosimilitud marginal:
$$BIC = log p(\cal{D} \mid \hat{\theta}) -\frac{dof(\hat{\theta})}{2} log {\text N} \approx \log p(\cal{D})$$
donde $dof(\hat{\theta})$ son los grados de liberdad del modelo y $\hat{\theta}$ es el estimador de máxima verosimilitud de $\theta$.

De esta manera el BIC se puede interpretar como una versión del logaritmo de la verosimilitud **penalizada** por la complejidad del modelo.

**Ejercicio 3:** Considere el modelo Beta-Binomial y que
las observaciones nos proporcionan como estadísticos $N_1=3$ y $N_0=17$.
Calcule la verosimilitud marginal y el estadístico BIC para seleccionar el mejor modelo,
dado que se considera conocido el valor del parámetro $b=2$. Explore valores para $a \in (0,20)$.

```R

```