```python
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

# Naive Baye's Classifier

Naive Baye's classifier is based on Baye's theorem and is used for classification problems. For instance in NLP text classification: topic modeling, sentiment analysis, spam detection etc

## Bayes' theorem:

Partition of set $X$ on subsets $\{A_i \subset X|I \in I\}$ is $X = \bigcup_{i \in I}A_i$ and $A_i \cap A_j = \emptyset$ for every pair $i,j \in I$

For each subset $A \subset X$ we have partition $X = A \cup A^c$

#### Total probability theorem:
For every partition $A_1, A_2 \dots, A_k$ of $\Omega$ and event $B \subset \Omega$:
$$P(B) = \sum_{i=1}^{k}P(B|A_i)P(A_i)$$

#### Theorem (Bayes' theorem):
Let $A_1, A_2 \dots, A_k$ be a partition of $\Omega$ such that $P(A_i) > 0$ for each $i \in \{1, 2, \dots, k\}$. Then for $B \subset \Omega$ event, such that $P(B) > 0$, for each $i \in \{1, 2, \dots, k\}$:
$$
P(A_i|B) = \frac{P(B|A_i)P(A_i)}{\sum_{j=1}^{k}P(B|A_j)P(A_j)}
$$

#### Note:
We call $P(A_i)$ the prior probability and $P(A_i|B)$ the posterior probability

For the events $A$ and $B$ such that $P(B) \gt 0$ we have:
$$
P(A|B) = \frac{P(B|A)P(A)}{P(B)}
$$

We can consider the partition of $\Omega$ on $A$ and $A^c$, the from Byes' theorem we have:
$$
P(A|B) = \frac{P(B|A)P(A)}{P(B|A)P(A) + P(B|A^c)P(A^c)} = \text{(by the total probability) }\frac{P(B|A)P(A)}{P(B)}
$$

#### Example:
Divide emails $A_1 = \text{"spam"}$, $A_2 = \text{"low priority"}$ and $A_3 = \text{"high priority"}$ and let: $P(A_1) = 0.7$, $P(A_2) = 0.2$ and $P(A_3) = 0.1$. ($P(A_1) + P(A_2) + P(A_3) = 0.7 + 0.2 + 0.1 = 1$)

Let $B$ be the event that email contains the word "free" and we know from previous experience that: $P(B|A_1) = 0.9$, $P(B|A_2) = 0.01$ and $P(B|A_3) = 0.01$.

If we receive the email with word "free" in it, what is the probability, that this email is spam?
From Bayes' theorem:
$$
P(A_1|B) = \frac{P(B|A_1)P(A_1)}{P(B|A_1)P(A_1) + P(B|A_2)P(A_2) + P(B|A_3)P(A_3)} = \frac{0.9 \cdot 0.7}{0.9 \cdot 0.7 + 0.01 \cdot 0.2 + 0.01 \cdot .01} = 0.995
$$

## Multi-dimensional case

#### Example Golf and Weather

```python
import pandas as pd
import numpy as np
from pathlib import Path
```

```python
path = Path('data')
nb = path / 'naive-bayes'
golf_csv = nb / 'golf.csv'
```

```python
def strip_txt(txt:str) -> str:
    return txt.replace("'", '').strip() if txt else txt
```

```python
df = pd.read_csv(golf_csv, converters={'outlook':strip_txt,
                                       'temp': strip_txt,
                                       'humidity': strip_txt,
                                       'wind': strip_txt,
                                       'label': strip_txt})
df
```

      outlook
      temp
      humidity
      wind
      label

      0
      Sunny
      Hot
      High
      Weak
      No

      1
      Sunny
      Hot
      High
      Strong
      No

      2
      Overcast
      Hot
      High
      Weak
      Yes

      3
      Rain
      Mild
      High
      Weak
      Yes

      4
      Rain
      Cool
      Normal
      Weak
      Yes

      5
      Rain
      Cool
      Normal
      Strong
      No

      6
      Overcast
      Cool
      Normal
      Strong
      Yes

      7
      Sunny
      Mild
      High
      Weak
      No

      8
      Sunny
      Cool
      Normal
      Weak
      Yes

      9
      Rain
      Mild
      Normal
      Weak
      Yes

      10
      Sunny
      Mild
      Normal
      Strong
      Yes

      11
      Overcast
      Mild
      High
      Strong
      Yes

      12
      Overcast
      Hot
      Normal
      Weak
      Yes

      13
      Rain
      Mild
      High
      Strong
      No

Let's calculate each Label probabilities

$P(\text{"Yes"}) = 9/14$ and $P(\text{"No"}) = 5/14$

Let't conditional probability of outlook 'Sunny' feature with respect of labels

$P(\text{"Yes"}|\text{"Sunny"}) = 2/9$ and $P(\text{"No"}|\text{"Sunny"}) = 3/5$

Let't conditional probability of outlook 'Overcast' feature with respect of labels

$𝑃("Yes"|"Overcast")=3/9$  and  $𝑃("No"|"Overcast")=0/5$

```python
df.outlook
```

    0        Sunny
    1        Sunny
    2     Overcast
    3         Rain
    4         Rain
    5         Rain
    6     Overcast
    7        Sunny
    8        Sunny
    9         Rain
    10       Sunny
    11    Overcast
    12    Overcast
    13        Rain
    Name: outlook, dtype: object

```python
y_vals = df[df.label.str.contains('Yes')].count()[0]
n_vals = df[df.label.str.contains('No')].count()[0]
f_vals = df.count()[0]
y_vals, n_vals, f_vals
```

    (9, 5, 14)

```python
df.outlook.unique()
```

    array(['Sunny', 'Overcast', 'Rain'], dtype=object)

```python
sunny_y = df[df.outlook.str.contains('Sunny') & df.label.str.contains('Yes')].count()[0]
sunny_n = df[df.outlook.str.contains('Sunny') & df.label.str.contains('No')].count()[0]
overcast_y = df[df.outlook.str.contains('Overcast') & df.label.str.contains('Yes')].count()[0]
overcast_n = df[df.outlook.str.contains('Overcast') & df.label.str.contains('No')].count()[0]
rain_y = df[df.outlook.str.contains('Rain') & df.label.str.contains('Yes')].count()[0]
rain_n = df[df.outlook.str.contains('Rain') & df.label.str.contains('No')].count()[0]
print(f'sunny_y = {sunny_y}/{y_vals}, sunny_n = {sunny_n}/{n_vals}')
print(f'overcast_y = {overcast_y}/{y_vals}, overcast_n = {overcast_n}/{n_vals}')
print(f'rain_y = {rain_y}/{y_vals}, rain_y = {rain_y}/{n_vals}')
```

    sunny_y = 2/9, sunny_n = 3/5
    overcast_y = 4/9, overcast_n = 0/5
    rain_y = 3/9, rain_y = 3/5

```python
def count_feat(col_name:str, col_val:str) -> int:
    return df[df[col_name].str.contains(col_val)].count()[0]

def count_cond(col_name:str, col_val:str, lab:str) -> int:
    return df[df[col_name].str.contains(col_val) & df.label.str.contains(lab)].count()[0]

def count_probs(col_name:str) -> int:
    col_vals = df[col_name].unique()
    for col_val in col_vals:
        val_y = count_cond(col_name, col_val, 'Yes')
        val_n = count_cond(col_name, col_val, 'No')
        val_f = count_feat(col_name, col_val)
        yield val_y, val_n, val_f, col_val

```

```python
col_vals = df.temp.unique()
temp_vals = [(ys, ns, fs, vls) for (ys, ns, fs, vls) in count_probs('temp')]
temp_vals
```

    [(2, 2, 4, 'Hot'), (4, 2, 6, 'Mild'), (3, 1, 4, 'Cool')]

```python
col_vals = [(col_name, [(ys, ns, fs, vls) for (ys, ns, fs, vls) in count_probs(col_name)])
            for col_name in df.columns]
col_vals
```

    [('outlook', [(2, 3, 5, 'Sunny'), (4, 0, 4, 'Overcast'), (3, 2, 5, 'Rain')]),
     ('temp', [(2, 2, 4, 'Hot'), (4, 2, 6, 'Mild'), (3, 1, 4, 'Cool')]),
     ('humidity', [(3, 4, 7, 'High'), (6, 1, 7, 'Normal')]),
     ('wind', [(6, 2, 8, 'Weak'), (3, 3, 6, 'Strong')]),
     ('label', [(0, 5, 5, 'No'), (9, 0, 9, 'Yes')])]

```python
lns = ''
for col_val in col_vals:
    ln = f'{col_val[0]}: \n' + '\n'.join(f'P({nm}) = {f_v}, P({nm}|Yes) = {y_v}/{y_vals}, P({nm}|No) = {n_v}/{n_vals}'
                               for y_v, n_v, f_v, nm in col_val[1]) + '\n'
    lns += ln
    lns += '===============\n'
print(lns)
```

    outlook:
    P(Sunny) = 5, P(Sunny|Yes) = 2/9, P(Sunny|No) = 3/5
    P(Overcast) = 4, P(Overcast|Yes) = 4/9, P(Overcast|No) = 0/5
    P(Rain) = 5, P(Rain|Yes) = 3/9, P(Rain|No) = 2/5
    ===============
    temp:
    P(Hot) = 4, P(Hot|Yes) = 2/9, P(Hot|No) = 2/5
    P(Mild) = 6, P(Mild|Yes) = 4/9, P(Mild|No) = 2/5
    P(Cool) = 4, P(Cool|Yes) = 3/9, P(Cool|No) = 1/5
    ===============
    humidity:
    P(High) = 7, P(High|Yes) = 3/9, P(High|No) = 4/5
    P(Normal) = 7, P(Normal|Yes) = 6/9, P(Normal|No) = 1/5
    ===============
    wind:
    P(Weak) = 8, P(Weak|Yes) = 6/9, P(Weak|No) = 2/5
    P(Strong) = 6, P(Strong|Yes) = 3/9, P(Strong|No) = 3/5
    ===============
    label:
    P(No) = 5, P(No|Yes) = 0/9, P(No|No) = 5/5
    P(Yes) = 9, P(Yes|Yes) = 9/9, P(Yes|No) = 0/5
    ===============

```python
model_vals = {col_name: {vls: (ys, ns, fs) for (ys, ns, fs, vls) in count_probs(col_name)}
            for col_name in df.columns if col_name != 'label'}
model_vals
```

    {'outlook': {'Sunny': (2, 3, 5), 'Overcast': (4, 0, 4), 'Rain': (3, 2, 5)},
     'temp': {'Hot': (2, 2, 4), 'Mild': (4, 2, 6), 'Cool': (3, 1, 4)},
     'humidity': {'High': (3, 4, 7), 'Normal': (6, 1, 7)},
     'wind': {'Weak': (6, 2, 8), 'Strong': (3, 3, 6)}}

Outlook: "Sunny",
Temperature: "Cool",
Humidity: "High",
Wind: "Strong"

```python
out_v = model_vals['outlook']['Sunny']
tmp_v = model_vals['temp']['Cool']
hum_v = model_vals['humidity']['High']
wnd_v = model_vals['wind']['Strong']
yes_raw = (out_v[0] / y_vals) * (tmp_v[0] / y_vals) * (hum_v[0] / y_vals) * (wnd_v[0] / y_vals) * (y_vals / f_vals)
no_raw = (out_v[1] / n_vals) * (tmp_v[1] / n_vals) * (hum_v[1] / n_vals) * (wnd_v[1] / n_vals) * (n_vals / f_vals)
yes_raw, no_raw
```

    (0.005291005291005291, 0.02057142857142857)

```python
p_x = (out_v[2] / f_vals) * (tmp_v[2] / f_vals) * (hum_v[2] / f_vals) * (wnd_v[2] / f_vals)
p_x
```

    0.021865889212827987

```python
yes_pred = yes_raw / p_x
no_pred = no_raw / p_x
print(f'yes_pred = {yes_pred}, no_pred = {no_pred}')
```

    yes_pred = 0.2419753086419753, no_pred = 0.9408

```python
def predict(x:tuple) -> tuple:
    out_v = model_vals['outlook'][x[0].capitalize()]
    tmp_v = model_vals['temp'][x[1].capitalize()]
    hum_v = model_vals['humidity'][x[2].capitalize()]
    wnd_v = model_vals['wind'][x[3].capitalize()]
    yes_raw = (out_v[0] / y_vals) * (tmp_v[0] / y_vals) * (hum_v[0] / y_vals) * (wnd_v[0] / y_vals) * (y_vals / f_vals)
    no_raw = (out_v[1] / n_vals) * (tmp_v[1] / n_vals) * (hum_v[1] / n_vals) * (wnd_v[1] / n_vals) * (n_vals / f_vals)
    p_x = (out_v[2] / f_vals) * (tmp_v[2] / f_vals) * (hum_v[2] / f_vals) * (wnd_v[2] / f_vals)
    yes_pred = yes_raw / p_x
    no_pred = no_raw / p_x

    return yes_pred, no_pred
```

```python
x_vec = ('Sunny', 'Cool', 'High', 'Strong')
y_pr, n_pr = predict(x_vec)
print(f'yes_pred = {y_pr}, no_pred = {n_pr}')
```

    yes_pred = 0.2419753086419753, no_pred = 0.9408

## Analysis of algorithm

Learning a Naive Bayes classifier is just a matter of counting how many times each attribute co-occurs with each class

As you observed algorithm fits more for categorical variables rather than numerical. Also if sample distribution is representative.

#### Advantages
- It performs better on multi-class classification.
- It performs better on smaller datasets.
- If independence is hold, Naive Baye's performs better than Logistic regression on smaller datasets.
- It performs better for categorical variables, for numerical variables Gaussian distribution is preffered.

#### Disadvantages
- If categorical variable is unseen during the training then we get zero frequency problem and smoothing technique is used (Laplasian Smoothing)
- It is known as bad estimator, probabilities should not be tacken in consideration
- In real life features are not independent

Main question is how can we deal with zero frequencies. Laplassian smoothing is a little trick:
$$P(x_i|C_p) = \frac{count(x_i, C_p) + 1}{\sum_{j=1}^{n}(count(x_j, C_p) + 1)}$$

or
$$P(x_i|C_p) = \frac{count(x_i, C_p) + 1}{\sum_{j=1}^{n}count(x_j, C_p) + n}$$

## Theoretical Part

Recall conditional probability
$$p(C_k \mid x_1, \dots, x_n)$$
and Baye's theorem
$$p(C_k \mid \mathbf{x}) = \frac{p(C_k) \ p(\mathbf{x} \mid C_k)}{p(\mathbf{x})}$$

Recall that for given $C$ the probability $$P(A \mid C)$$ can be considered as usual probability mesure:
- $$P(\Omega \mid C) = 1$$
- $$P(A \mid C) \le 1$$
etc

The evens $A$ and $B$ are independent if $P(A\cap B) = P(A)P(B)$ or $P(A, B) = P(A)P(B)$ we can use this formula for conditional independence (implies from ebove property of conditional probability)
$$P(A \cap B | C) = P(A \mid C)P(B \mid C)$$

By the definition of conditional probability $P(A\cap B)= P(A \mid B)P(B)$

$$\begin{align}
P(A \mid B \cap C) = \frac{P(B \cap C \cap A)}{P(B \cap C)}= \frac{P(B \cap A \cap C)}{P(B \cap C)} = \\
\frac{P(A \cap B \mid C)P(C)}{P(B \cap C)} = \\
\frac{P(A \mid C)P(B \mid C)P(C)}{P(B \cap C)} = \\
\frac{P(A \mid C)P(B \mid C)P(C)}{P(B \mid C)P(C)} = P(A \mid C)
\end{align}$$

$$P(A| B \cap C) = P(A \mid B, C) = P(A\mid C)$$

$$\begin{align}
P(C_k, x_1, \dots, x_n) & = P(x_1, \dots, x_n, C_k) \\
                        & = P(x_1 \mid x_2, \dots, x_n, C_k) \ P(x_2, \dots, x_n, C_k) \\
                        & = P(x_1 \mid x_2, \dots, x_n, C_k) \ P(x_2 \mid x_3, \dots, x_n, C_k) \ P(x_3, \dots, x_n, C_k) \\
                        & = \dots \\
                        & = P(x_1 \mid x_2, \dots, x_n, C_k) \ P(x_2 \mid x_3, \dots, x_n, C_k) \dots   P(x_{n-1} \mid x_n, C_k) \ p(x_n \mid C_k) \ P(C_k) \\
\end{align}$$

In general
$$P(x_i \mid x_{i+1}, \dots ,x_{n}, C_k ) = P(x_i \mid C_k)$$

We have formula:
$$\begin{align}
P(C_k, x_1, \dots, x_n) & = \frac{P(x_1 \mid C_k)P(x_2 \mid C_k) \dots P(x_n \mid C_k)P(C_k)}{P(x)} \\
& = \frac{P(x_1 \mid C_k)P(x_2 \mid C_k) \dots P(x_n \mid C_k)P(C_k)}{P(x_1, x_2, \dots, x_n)} \\
&= \frac{P(x_1 \mid C_k)P(x_2 \mid C_k) \dots P(x_n \mid C_k)P(C_k)}{P(x_1)P(x_2) \dots P(x_n)} \\
\end{align}$$

Denominator is the same for all classes so classifier will decide by the formula:
$$\hat{y} = \underset{k \in \{1, \dots, K\}}{\operatorname{argmax}} \ P(C_k) \displaystyle\prod_{i=1}^n P(x_i \mid C_k)$$

## Different types of Naive Baye's

#### Multinomial naive Bayes

Let event occures with $(p_1, \dots, p_n)$ probabilities, then we have:

$$P(\mathbf{x} \mid C_k) = \frac{(\sum_i x_i)!}{\prod_i x_i !} \prod_i {p_{ki}}^{x_i}$$

Then our estimator with $\log$ becomes linear classifier:
$$
\begin{align}
\log p(C_k \mid \mathbf{x}) & \varpropto \log \left( p(C_k) \prod_{i=1}^n {p_{ki}}^{x_i} \right) \\
                       & = \log p(C_k) + \sum_{i=1}^n x_i \cdot \log p_{ki}                 \\
                       & = b + \mathbf{w}_k^\top \mathbf{x}
\end{align}
$$

where $b = \log p(C_k)$ and $w_{ki} = \log p_{ki}$

#### Bernoulli naive Bayes

When we have occurrence instead of frequency:
$$P(\mathbf{x} \mid C_k) = \prod_{i=1}^n p_{ki}^{x_i} (1 - p_{ki})^{(1-x_i)}$$

#### Gaussian naive Bayes

Let for some $x$ and class $C_k$ data is distrubuted with normal (Gaussian) distribution.
Compute mean and variance of $x$ for each class: $\mu_k$ and $\sigma_{k}^{2}$, then fop value $v$:
$$p(x=v \mid C_k)=\frac{1}{\sqrt{2\pi\sigma^2_k}}\,e^{ -\frac{(v-\mu_k)^2}{2\sigma^2_k} }$$

#### Excercise:
Try The approach for spam classifier

#### Excercise:
Classify Iris dataset with Naive Baye's classifier

#### Excercise:
Classify Wine dataset with Naive Baye's classifier

#### Excercise:
Classify Adult dataset with Naive Baye's classifier


# Learning from data: Hypothesis testing and Bayesian model Selection

**Christian Forssén**, Department of Physics, Chalmers University of Technology, Sweden

Date: **Oct 1, 2019**

Copyright 2018-2019, Christian Forssén. Released under CC Attribution-NonCommercial 4.0 license

So far, we have been concerned with the problem of parameter estimation. In studying the linear relationship between two quantities, for example, we discussed how to infer the slope and the offset of the associated straight-line model. Often, however, there is a question as to whether a quadratic, or even cubic, function might be a more appropriate model. In this lecture, we will consider the broad class of scientific problems when there is uncertainty as to which one of a set of alternative models is most suitable. In the Bayesian terminology these can be labeled as **Model Selection** problems and we will discuss them in some depth.

One of the main objectives in science is that of inferring the truth of one or more hypotheses about how some aspect of nature works. Because we are always in a state of incomplete information, we can never prove any hypothesis (theory) is true.

We will start, however, with a brief discussion on sampling theory and the frequentist approach to **hypothesis testing**. This will involve the introduction of the $P$-value or significance measure---quantities that are often misinterpreted even by scientists themselves. See, for example, the following comment published in Nature (March 20, 2019): [Scientists rise up against statistical significance](https://www.nature.com/articles/d41586-019-00857-9).

# Frequentist hypothesis testing
Recall that in frequentist statistics, probability statements are restricted to random variables. A hypothesis can not be considered a random variable, and therefore we are restricted to a much more indirect approach when trying to infer its truth, or rather when attempting to falsify it.

## Basic idea

The standard sampling theory approach to hypothesis testing is to construct a statistical test. The basic idea is the following:

**Frequentist hypothesis testing.**

The sampling theory hypothesis test is designed to compare a selected statistic from the measured data with expected results from a very large number of hypothetical repeated measurements under the assumption that a chosen null hypothesis ($\mathcal{H}_0$) is true.

* The null hypothesis is accepted or rejected purely on the basis of how unexpected the data were to $\mathcal{H}_0$, not on how much better the alternative hypothesis ($\mathcal{H}_A$) predicted the data.

* The degree of ''unexpectedness'' is based on a statistic, such as the sample mean or the $\chi^2$ statistic.

* The statistic is a random variable and it is chosen so that its distribution can be easily computed given the truth of the null hypothesis. In other words, this is the distribution of the chosen statistic for a very large number of hypothetical repeated measurements under the assumption that the null hypothesis is true.

* This statistic is then computed for the observed data set and its value is compared with the distribution that is associated with the truth of the null hypothesis.

* If the statistic from the observed data falls in a very unlikely spot on this distribution (the threshold is to be defined beforehand) we choose to reject the null hypothesis at some confidence level on the basis of the measured data set.

### Hypothesis testing with the chi-squared statistic

A very common statistic to use is the $\chi^2$ measure. A good example is found in Gregory, ch 7.2.1, with the measurements of flux density from a distant galaxy over a period of 6000 days. The main steps of the presented analysis are the following:

* Choose as a null hypothesis that the galaxy has an unknown, but constant, flux density. If we can reject this hypothesis at e.g. the 95% confidence level, then this provides indirect evidence(?) for the alternative hypothesis that the radio emission is variable.

* In this example, it is assumed that the measurement errors are independent and identically distributed (**iid**) according to a normal distribution with a fixed standard deviation $\sigma$ that is known beforehand.

* The $\chi^2$ statistic from the data set is evaluated ($x_i$ is the data and $\bar{x}$ is the average from the sample)

$$
\chi^2 = \sum_{i=1}^n \frac{(x_i - \bar{x})^2}{\sigma^2}.
$$

* In our example we had 15 data points, but we are using them first to estimate the mean $\mu$. Therefore, we loose one degree of freedom and are left with 14. This number will determine the form of the $\chi^2$ distribution that will be used for comparison with our actual $\chi^2$ statistic.

* The question of how unlikely is this value of $\chi^2$ is by convention interpreted in terms of the area in the tail of the $\chi^2$ distribution beyond this line. This is called the $P$-value or significance.

The $\chi^2$ distribution for 14 degrees of freedom. The value computed from the measurements of flux density from a galaxy is indicated by a vertical line. The shaded area is the $P$-value. It is 2% in this particular example so we would reject the null hypothesis with 98% confidence. (Gregory, Fig. 7.2)

At the point of performing this comparison, and making a final statement, the sampling theory school divides itself into two camps:

1. One camp uses the following protocol: first, before looking at the data, pick the significance level of the test (e.g. 5%), and determine the critical value of $\chi^2$ above which the null hypothesis will be rejected. (The significance level is the fraction of times that the statistic $\chi^2$ would exceed the critical value, if the null hypothesis were true.) Then, compare the actual $\chi^2$ with the critical value, and declare the outcome of the test, and its significance level (which was fixed beforehand).

2. The second camp looks at the data, finds $\chi^2$, then looks in the table of $\chi^2$-distributions for the significance level, $P$, for which the observed value of $\chi^2$ would be the critical value. The result of the test is then reported by giving this value of $p$, which is the fraction of times that a result as extreme as the one observed, or more extreme, would be expected to arise if the null hypothesis were true.

# Bayesian model selection

## The Story of Dr. A and Prof. B
[Reproduced, with some modifications, from Sivia, 2006].
> *Dr. A has a theory; Prof. B also has a theory, but with an adjustable parameter $\lambda$. Whose theory should we prefer on the basis of data $D$?*--- Jefferys (1939), Gull (1988), Sivia (2006)

It is clear that we need to evaluate the posterior probabilities for A and B being correct to ascertain the relative merit of the two theories. If the ratio of the posterior probabilities,

$$
\begin{equation}
\text{posterior ratio} = \frac{p(A |D, I )}{p(B|D,I)}
\label{eq:sivia_41} \tag{1}
\end{equation}
$$

is very much greater than one, then we will prefer A’s theory; if it is very much less than one, then we prefer that of B; and if it is of order unity, then the current data are insufficient to make an informed judgement.

To estimate the odds, let us start by applying Bayes’ theorem to both the numerator and the denominator; this gives

$$
\begin{equation}
\frac{p(A|D,I)}{p(B|D,I)} = \frac{p(D|A,I) p(A|I)}{p(D|B,I) p(B|I)}
\label{eq:sivia_42} \tag{2}
\end{equation}
$$

because the term $p(D|I)$ cancels out, top and bottom. As usual, probability theory warns us immediately that the answer to our question depends partly on what we thought about the two theories before the analysis of the data. To be fair, we might take the ratio of the prior terms, on the far right of Eq. ([2](#eq:sivia_42)), to be unity; a harsher assignment could be based on the past track records of the theorists! To assign the probabilities involving the experimental measurements, $p(D|A,I)$ and $p(D|B,I)$, we need to be able to compare the data with the predictions of A and B: the larger the mismatch, the lower the corresponding probability. This calculation is straightforward for Dr A, but not for Prof B; the latter cannot make predictions without a value for $\lambda$.

To circumvent this difficulty, we can use the sum and product rule to relate the probability we require to other pdfs which might be easier to assign. In particular, marginalization and the product rule allow us to express $p(D | B , I )$ as

$$
\begin{equation}
p(D|B,I) = \int d\lambda p(D,\lambda|B,I) =
\int d\lambda p(D|\lambda,B,I) p(\lambda|B,I).
\label{eq:sivia_43} \tag{3}
\end{equation}
$$

The first term in the integral $p(D | \lambda, B , I )$, where the value of $\lambda$ is given, is now just an ordinary likelihood function; as such, it is on a par with $p(D|A,I)$. The second term is B’s prior pdf for $\lambda$; the onus is, therefore, on the theorist to articulate his or her state of knowledge, or ignorance, before getting access to the data.

To proceed further analytically, let us make some simplifying approximations. Assume that, a priori, Prof B is only prepared to say that $\lambda$ must lie between the limits $\lambda_\mathrm{min}$ and $\lambda_\mathrm{max}$; we can then naively assign a uniform prior within this range:

$$
\begin{equation}
p(\lambda|B,I) = \frac{1}{\lambda_\mathrm{max}-\lambda_\mathrm{min}} \quad \text{for } \lambda_\mathrm{min} \leq \lambda \leq \lambda_\mathrm{max},
\label{eq:sivia_44} \tag{4}
\end{equation}
$$

and zero otherwise. Let us also take it that there is a value $\lambda_0$ which yields the closest agreement with the measurements; the corresponding probability $p(D|\lambda_0,B,I)$ will be the maximum of B’s likelihood function. As long as this adjustable parameter lies in the neighbourhood of the optimal value, $\lambda_0 \pm \delta\lambda$, we would expect a reasonable fit to the data; this can be represented by the Gaussian pdf

$$
\begin{equation}
p(D|\lambda,B,I) = p(D|\lambda_0,B,I) \exp \left[ − \frac{(\lambda−\lambda_0)^2}{2\delta\lambda^2} \right].
\label{eq:sivia_45} \tag{5}
\end{equation}
$$

The assignments of the prior ([4](#eq:sivia_44)) and the likelihood ([5](#eq:sivia_45)) are illustrated in Fig. [fig:sivia_41](#fig:sivia_41). We may note that, unlike the prior pdf $p(\lambda|B,I)$, B’s likelihood function need not be normalized with respect to $\lambda$; in other words, $p(D|\lambda_0,B,I)$ need not equal $1/ \delta\lambda \sqrt{2\pi}$ . This is because the $\lambda$ in $p(D|\lambda,B,I)$ appears in the conditioning statement, whereas the normalization requirement applies to quantities to the left of the ‘|’ symbol.

A schematic representation of the prior pdf (dashed line) and the likelihood function (solid line) for the parameter $\lambda$ in Prof B’s theory.

In the evaluation of $p(D | B , I )$, we can make use of the fact that the prior does not depend explicitly on $\lambda$; this enables us to take $p(\lambda|B,I)$ outside the integral in Eq. ([3](#eq:sivia_43))

$$
\begin{equation}
p(D|B,I) = \frac{1}{\lambda_\mathrm{max} - \lambda_\mathrm{min}} \int_{\lambda_\mathrm{min}}^{\lambda_\mathrm{max}} d\lambda
p(D|\lambda,B,I),
\label{eq:sivia_46} \tag{6}
\end{equation}
$$

having set the limits according to the specified range. Assuming that the sharp cut-offs
do not cause a significant truncation of the Gaussian pdf of the likelihood,
its integral will be equal to $\delta\lambda \sqrt{2\pi}$ times $p(D|\lambda_0,B,I)$. The troublesome term then reduces to
$$
p(D|B,I) = \frac{1}{\lambda_\mathrm{max} - \lambda_\mathrm{min}} p(D|\lambda_0,B,I) \delta\lambda \sqrt{2\pi}.
$$
Substituting this into Eq. ([2](#eq:sivia_42)), we finally see that the ratio of the posteriors required to answer our original question decomposes into the product of three terms:

$$
\begin{equation}
\frac{p(A|D,I)}{p(B|D,I)} =  \frac{p(A|I)}{p(B|I)} \times \frac{p(D|A,I)}{p(D|\lambda_0,B,I)} \times \frac{\lambda_\mathrm{max} - \lambda_\mathrm{min}}{\delta\lambda \sqrt{2\pi}}.
\label{eq:sivia_48} \tag{7}
\end{equation}
$$

The first term on the right-hand side reflects our relative prior preference for the alternative theories; to be fair, we can take it to be unity. The second term is a measure of how well the best predictions from each of the models agree with the data; with the added flexibility of his adjustable parameter, this maximum likelihood ratio can only favour B. The goodness-of-fit, however, cannot be the only thing that matters; if it was, we would always prefer more complicated explanations. Probability theory tells us that there is, indeed, another term to be considered. As assumed earlier in the evaluation of the marginal integral of Eq. ([3](#eq:sivia_43)), the prior range $\lambda_\mathrm{max} - \lambda_\mathrm{min}$ will generally be much larger than the uncertainty $\pm\delta\lambda$ permitted by the data. As such, the final term in Eq. ([7](#eq:sivia_48)) acts to penalize B for the additional parameter; for this reason, it is often called an Ockham factor. That is to say, we have naturally encompassed the spirit of Ockham’s Razor: *‘Frustra fit per plura quod potest fieri per pauciora’* or, in English, *‘it is vain to do with more what can be done with fewer’* .

Although it is satisfying to quantify the everyday guiding principle attributed to the thirteenth-century Franciscan monk William of Ockham (or Occam, in Latin), that we should prefer the simplest theory which agrees with the empirical evidence, we should not get too carried away by it. After all, what do we mean by the simpler theory if alternative models have the same number of adjustable parameters? In the choice between Gaussian and Lorentzian peak shapes, for example, both are defined by the position of the maximum and their width. All that we are obliged to do, and have done, in addressing such questions is to adhere to the rules of probability.

While accepting the clear logic leading to Eq ([7](#eq:sivia_48)), many people rightly worry about the question of the limits $\lambda_\mathrm{min}$ and $\lambda_\mathrm{max}$. Jeffreys (1939) himself was concerned and pointed out that there would be an infinite penalty for any new parameter if the range was allowed to go to $\pm\infty$. Stated in the abstract, this would appear to be a severe limitation. In practice, however, it is not generally such a problem: since the analysis is always used in specific contexts, a suitable choice can usually be made on the basis of the relevant background information. Even in uncharted territory, a small amount of thought soon reveals that our state of ignorance is always far from the $\pm\infty$ scenario. If $\lambda$ was the coupling constant (or strength) for a possible fifth force, for example, then we could put an upper bound on its magnitude because everybody would have noticed it by now if it had been large enough! We should also not lose sight of the fact that the precise form of Eq ([7](#eq:sivia_48)) stems from our stated simplifying approximations; if these are not appropriate, then eqns ([2](#eq:sivia_42)) and ([3](#eq:sivia_43)) will lead us to a somewhat different formula.

In most cases, our relative preference for A or B is dominated by the goodness of the fit to the data; that is to say, the maximum likelihood ratio in eqn ([7](#eq:sivia_48)) tends to overwhelm the contributions of the other two terms. The Ockham factor can play a crucial role, however, when both theories give comparably good agreement with the measurements. Indeed, it becomes increasingly important if B’s theory fails to give a significantly better fit as the quality of the data improves. In that case, $\delta\lambda$ continues to become smaller but the ratio of best-fit likelihoods remains close to unity; according to Eq. ([7](#eq:sivia_48)), therefore, A’s theory is favoured ever more strongly. By the same token, the Ockham effect disappears if the data are either few in number, of poor quality or just fail to shed new light on the problem at hand. This is simply because the posterior ratio of Eq. ([7](#eq:sivia_48)) is then roughly equal to the complementary prior one, since the empirical evidence is very weak; hence, there is no inherent preference for A’s theory unless it is explicitly encoded in $p(A|I)/p(B|I)$. This property can be verified formally by going back to Eqs. ([2](#eq:sivia_42)), ([5](#eq:sivia_45)) and ([6](#eq:sivia_46)), and considering the poor-data limit in which
$\delta\lambda \gg \lambda_\mathrm{max}-\lambda_\mathrm{min}$ and $p(D|\lambda_0,B,I) \approx p(D|A,I)$.

### One adjustable parameter each

Some further interesting features arise when we consider the case where Dr A also has one adjustable parameter; call it $\mu$. If we make the same sort of probability assignments, and simplifying approximations, as for Prof B, then we find that

$$
\begin{equation}
\frac{p(A|D,I)}{p(B|D,I)} =  \frac{p(A|I)}{p(B|I)} \times \frac{p(D|\mu_0,A,I)}{p(D|\lambda_0,B,I)} \times \frac{\delta\mu(\lambda_\mathrm{max} - \lambda_\mathrm{min})}{\delta\lambda(\mu_\mathrm{max} - \mu_\mathrm{min})}.
\label{eq:sivia_49} \tag{8}
\end{equation}
$$

This could represent the situation where we have to choose between a Gaussian and Lorentzian shape for a signal peak, but one associated parameter is not known. The position of the maximum may be fixed at the origin by theory, for example, and the amplitude constrained by the normalization of the data; A and B could then be the hypotheses favouring the alternative lineshapes, where $\delta\mu$ and $\delta\lambda$ are their related full-width-half-maxima. If we give equal weight to A and B before the analysis, and assign a similar large prior range for both $\mu$ and $\lambda$, then Eq. ([8](#eq:sivia_49)) reduces to
$$
\frac{p(A|D,I)}{p(B|D,I)} \approx  \frac{p(D|\mu_0,A,I)}{p(D|\lambda_0,B,I)} \times \frac{\delta\mu}{\delta\lambda}.
$$

For data of good quality, the dominant factor will tend to be the best-fit likelihood ratio. If both give comparable agreement with the measurements, however, then the shape with the larger error-bar for its associated parameter will be favoured. At first sight, it might seem rather odd that the less discriminating theory can gain the upper hand. It appears less strange once we realize that, in the context of model selection, a larger ‘error-bar’ means that more parameter values are consistent with the given hypothesis; hence its preferential treatment.

### One adjustable parameter each; different prior ranges

Finally, we can also consider the situation where Mr A and Mr B have the same physical theory but assign a different prior range for $\lambda$ (or $\mu$). Although Eq. ([7](#eq:sivia_48)) can be seen as representing the case when $(\mu_\mathrm{max} - \mu_\mathrm{min})$ is infinitesimally small, so that A has no flexibility, Eq. ([8](#eq:sivia_49)) is more appropriate when the limits set by both theorists are large enough to encompass all the parameter values giving a reasonable fit to the data. With equal initial weighting towards A and B, the latter reduces to
$$
\frac{p(A|D,I)}{p(B|D,I)} =  \frac{\lambda_\mathrm{max} - \lambda_\mathrm{min}}{\mu_\mathrm{max} - \mu_\mathrm{min}}.
$$
because the best-fit likelihood ratio will be unity (since $\lambda_0 = \mu_0$) and $\delta\lambda = \delta\mu$. Thus, our analysis will lead us to prefer the theorist who gives the narrower prior range; this is not unreasonable as he must have had some additional insight to be able to predict the value of the parameter more accurately.

## Comparison with parameter estimation
The dependence of the result in Eq. ([7](#eq:sivia_48)) on the prior range $(\lambda_\mathrm{max} - \lambda_\mathrm{min})$ can seem a little strange, since we haven’t encountered such behaviour in the preceding chapters. It is instructive, therefore, to compare the model selection analysis with parameter estimation. To infer the value of $\lambda$ from the data, given that B’s theory is correct, we use Bayes’ theorem:

$$
\begin{equation}
p(\lambda|D,B,I) = \frac{p(D|\lambda,B,I) p(\lambda|B,I)}{p(D|B,I)}.
\label{eq:sivia_410} \tag{9}
\end{equation}
$$

The numerator is the familiar product of a prior and likelihood, and the denominator is usually omitted since it does not depend explicitly on $\lambda$; hence this relationship is often written as a proportionality. From the story of Dr A and Prof B, however, we find that the neglected term on the bottom plays a crucial role in ascertaining the merit of B’s theory relative to a competing alternative.

* In recognition of its new-found importance, the denominator in Bayes’ theorem is sometimes called the **‘evidence’** for B; it is also referred to as the ‘marginal likelihood’, the ‘global likelihood’ and the ‘prior predictive’.

* Since all the components necessary for both parameter estimation and model selection appear in Eq. ([9](#eq:sivia_410)), we are not dealing with any new principles; the only thing that sets them apart is that we are asking different questions of the data.

A simple way to think about the difference between parameter estimation and model selection is to note that, to a good approximation, the former requires the location of the maximum of the likelihood function whereas the latter entails the calculation of its average value. As long as $\lambda_\mathrm{min}$ and $\lambda_\mathrm{max}$ encompass the significant region of $p(D|\lambda,B,I)$ around $\lambda_0$, the precise bounds do not matter for estimating the optimal parameter and need not be specified. Since the prior range defines the domain over which the mean likelihood is computed, due thought is necessary when dealing with model selection. Indeed, it is precisely this act of comparing ‘average’ likelihoods rather than ‘maximum’ ones which introduces the desired Ockham balance to the goodness- of-fit criterion. Any likelihood gain from a better agreement with the data, allowed by the greater flexibility of a more complicated model, has to be weighed against the additional cost of averaging it over a larger parameter space.

# Evidence calculations
The actual computation of Bayesian evidences can be a challenging task. Recall that we often have knowledge of the posterior distribution only through sampling. In many cases, the simple Laplace method can be used to compute the evidence approximately, while in other cases we have to rely on special sampling algorithms such as nested sampling or parallel tempering with thermodynamic integration.

## Laplace's method
The idea behind the Laplace approximation is simple. We assume that an unnormalized probability density $P^*(\theta)$ has a peak at a point $\theta_0$. We are interested in the evidence, $Z_P$, which is given by the normalizing constant
$$
Z_P = \int P^*(\theta) d^K\theta,
$$
where we consider the general case in which $\theta$ is in a $K$-dimensional space.

We Taylor-expand the logarithm $\log P^*$ around the peak:
$$
\log P^*(\theta) = \log P^*(\theta_0) - \frac{1}{2} (\theta - \theta_0)^T \Sigma^{-1} (\theta - \theta_0) + \ldots,
$$
where $\Sigma^{-1} = H$ is the (Hessian) matrix of second derivatives at the maximum
$$
H_{ij} = - \left. \frac{\partial^2}{\partial \theta_i \partial \theta_j}  \log P^*(\theta)\right|_{\theta=\theta_0}.
$$

We then approximate $P^*(\theta)$ by an unnormalized Gaussian,
$$
Q^*(\theta) \equiv P^*(\theta_0) \exp \left[ - \frac{1}{2}(\theta - \theta_0)^T \Sigma^{-1} (\theta - \theta_0) \right],
$$
and we approximate the normalizing constant $Z_P$ by the normalizing constant
of this Gaussian,
$$
Z_P \approx Z_Q = P^*(\theta_0) \sqrt{\frac{(2\pi)^K}{\det\Sigma^{-1}}}.
$$
Predictions can then be made using this approximation $Q$. Physicists also call this widely-used approximation the saddle-point approximation.

Note, in particular, that if we consider a chi-squared pdf: $P^*(\theta) = \exp \left( -\frac{1}{2} \chi^2(\theta)\right)$, then we get
$$
Z_P \approx \exp \left( -\frac{1}{2} \chi^2(\theta_0)\right) \sqrt{\frac{(4\pi)^K}{\det\Sigma^{-1}}},
$$
where there is a factor $2^{K/2}$ that comes from the extra factor $1/2$ multiplying the covariance matrix $\Sigma^{-1}$ and therefore appearing in all $K$ eigenvalues.

The fact that the normalizing constant of a Gaussian is given by
$$
\int d^K\theta \exp \left[ - \frac{1}{2}\theta^T \Sigma^{-1} \theta \right] = \sqrt{\frac{(2\pi)^K}{\det\Sigma^{-1}}},
$$
can be proved by making an orthogonal transformation into the basis $u$ in which $\Sigma$ is transformed into a diagonal matrix. The integral then separates into a product of one-dimensional integrals, each of the form
$$
\int du_i \exp \left[ -\frac{1}{2} \lambda_i u_i^2 \right] = \sqrt{\frac{2\pi}{\lambda_i}}
$$
The product of the eigenvalues $\lambda_i$ is the determinant of $\Sigma^{-1}$.

Note that the Laplace approximation is basis-dependent: if $\theta$ is transformed to a nonlinear function $u(\theta)$ and the density is transformed to $P(u) = P(\theta) |d\theta/du|$ then in general the approximate normalizing constants $Z_Q$ will be different. This can be viewed as a defect---since the true value $Z_P$ is basis-independent in this approximation---or an opportunity, because we can hunt for a choice of basis in which the Laplace approximation is most accurate.