## Connecting Naive Bayes and Logistic Regression

### 1. Basics of Probability and Likelihood

Each probability distribution has a probability density function (PDF) for continuous distribution e.g. gaussian distribution (or a probability mass function or PMF for discrete distribution, e.g. binomial distribution) which indicates the probability of a sample (some point) taking a particular value. This function is usually denoted by $P(y|\theta)$ where $y$ is the value of the sample and $\theta$ is the parameter that describes the PDF/PMF. When more than 1 sample is drawn independently of one another then we can write as ---

$P(y_1, y_2, ...., y_n|\theta) = \prod \limits_{i=1}^n P(y_i|\theta)\, ...... (1.1)$

----------------------------------------------
**Likelihood**
----------------------------------------------

We consider the PDF's in calculations when we know the distribution (and corresponding parameter $\theta$) and want to deduce $y_i$'s. Here we think of $\theta$ is fixed (known) for different samples we want to deduce different $y_i$'s. The likelihood function is same but with a twist. Here $y_i$'s are known and $\theta$ is the variable that we want to determine. We can think of this as collecting data samples $(y_i)$ and want to find a distribution (with parameter $\theta$).

$\underbrace {L(y_1, ...., y_n|\theta)}_\text{$\theta$ unknown, $y_i's$ known} \, \, = \underbrace {P(y_1, ...., y_n|\theta)}_\text{$\theta$ fixed/known, $y_i's$ change/unknown}$

----------------------------------------------
**Bernoulli & Normal Distributions**
----------------------------------------------

Bernoulli distribution is a discrete distribution, takes value 1 with probability $\theta$ and 0 with probability $1-\theta$. If $P(y|\theta)$ be the probability that the event returns value $y$ given parameter $\theta$. Then--

$$L(y|\theta) = P(y|\theta) = \begin{cases} \theta,  & \text{if}\, y=1 \\ 1-\theta , & \text{if}\, y=0\end{cases}\, ...... (1.2)$$

They can be combined in a simpler form $P(y|\theta) = \theta ^y (1-\theta) ^{1-y}$

Assuming independent samples we can write the likelihood as follows--

$L(y_1, y_2, ....., y_n|\theta) = \prod \limits_{i=1}^n  \theta ^{y_i}\,  (1-\theta) ^{1-y_i}\, ...... (1.3)$

I guess we can already see the basis of binary cross-entropy loss function.

We can write the same for Normal distribution too....

$L(y_1, y_2, ....., y_n|\theta) = \prod \limits_{i=1}^n \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left( - \frac{1}{2 \sigma^2} (y_i - \mu)^2 \right)\, ...... (1.4)$

This also should tell us about the basis of mean squared error for regression problems.

----------------------
**MLE (Maximum Likelihood Estimation)**
------------------------

MLE is closely related with modelling data; If we observe data points $y_1, y_2, ...., y_n$ and assume that it is from a distribution parametrized by $\theta$, then likelihood is given by $L(y_1, y_2, ...., y_n|\theta)$, then $\theta _{\text{MLE}}$ of the parameter $\theta$ is the value that maximises the likelihood $L(y_1,…,y_n|\theta)$.

For independent observations we can write as below

$$\begin{align} \theta _{\text{MLE}} &= \text{arg}\,  \max _{\theta}\, L(y_1,…,y_n|\theta) \\ &= \text{arg}\,  \max _{\theta}\, \text{log}\, L(y_1,…,y_n|\theta)\, \, ; \text{maximising}\,  f\equiv \text{maximising log} f\\ &=\text{arg}\,  \max _{\theta}\, \text{log}\, \prod \limits_{i=1}^n L(y_i|\theta) \\ &= \text{arg}\,  \max _{\theta}\, \sum \text{log}\, L(y_i|\theta) \, ...... (1.5)\end{align}$$

Convention for optimization is minimizing a function; thus maximizing the likelihood boils down to minimizing the negative log likelihood.

$$\theta_{\text{MLE}} =  \text{arg} \min _{\theta} \, \text{NLL}\, (y_1,…,y_n|\theta) ; \, \, \text{NLL}\, (y_1,…, y_n|\theta)=−\sum \limits_{i=1}^{n} \text{log} L(y_i|\theta) \, ...... (1.6)$$

----------------------------------
_Training a Neural-Net_
--------------------------------

For training a neural net (MLE) we can think of optimizing the all the weights $w_i$'s so that it maximizes the likelihood of observing the data. Let's think about a neural network NN with weights $w$. Let's, assume $x_i$ is some data point, (e.g. an image to be classified, or an $x$ value for which we want to predict the $y$ value). The neural network prediction (the feedforward value) $\hat{𝑦}_i $ is

$\hat{y}_i = \text{NN}\, (x_i|w)\, ...... (1.7)$

Training a neural network is about optimizing $w$. Suppose we have some training data consisting of inputs and the associated labels. Let the data be $x_i$ and the labels $y_i$ for $i=1,…,n$, where $n$ is the number of training samples.  So training data $D \equiv \{(x_1, y_1), \, (x_2, y_2), \, ....\, (x_i, y_i)\}$ consists of data-label pairs. The weights of the trained neural network are then those that minimise the negative log-likelihood

$$\begin{align} w^* &= \text{arg}\, \min_{w} \left( - \sum\limits_{i=1}^n \text{log}\, L(y_i|\hat{y_i})\right)\\ &= \text{arg}\, \min_{w} \left(- \sum \limits_{i=1}^n \text{log} L\left(y_i|\text{NN}\, (x_i|w) \right)\right)\end{align}$$.

In practice, determining the true optimum $w^∗$ is not always possible. Instead, an approximate value is sought using gradient descent method, usually via a backpropagation of derivatives and some optimization algorithm such as RMSprop or Adam.

### 2. Naive Bayes Classifier

Naive Bayes Classifier is an example of Generative classifier while Logistic Regression is an example of discriminative classifier.

**Discriminative Classifier:** In general a classification problem can simply be thought of as predicting a class label for a given input vector $p(C_k|x)$. In discriminative model we assume some functional form for $p(C_k|x)$ and estimate parameters directly from training data.

**Generative Classifier:** In generative model we estimate the parameters of $p(x|C_k)$ i.e. probability distribution of the inputs for each class along with the class priors $p(C_k)$. Both of them are used in Bayes' theorem to calculate $p(C_k|x)$.

**Bayes Theorem:** Bayes' theorem in this context can be written as--

$$\begin{align} p(C_k|x) = \frac{p(x|C_k) \, p(C_k)}{p(x)}\end{align} ...... (2.1)$$

$p(x)$ can be thought of as a normalizing constant; $p(x) = \sum _k \, p(x|C_k) \, p(C_k)$

Let's consider a generalized scenario where our data have $d$ features and $K$ classes then the equation above can be written as ---

$$\begin{align} p(Y = c_k| X_1, X_2, \ldots, X_d) = \frac{p(X_1,X_2\ldots,X_d | Y=c_k)\, p(Y=c_k)}{\sum_{k=1}^K p(X_1,\ldots,X_d | Y=c_k)\, p(Y=c_k)}\end{align}\, ...... (2.2)$$

Class conditional probability term can be greatly simplified by assuming 'naively' that each of the data features $X_i$'s are conditionally independent of each other given the class $Y$.

Now we rewrite the equation $(2.2)$, as follows--

$$\begin{align} p(Y = c_k| X_1, X_2, \ldots, X_d) = \frac{p(Y=c_k)\, \prod \limits_{i=1}^d \, p(X_i| Y=c_k)\, }{\sum_{k=1}^K  p(Y=c_k) \, \prod p(X_i| Y=c_k)\,}\end{align}\, ...... (2.3)$$

_This is the fundamental equation for Naive Bayes Classifier._

Once the class prior distribution and class-conditional densities are estimated, the Naive Bayes classifier model can then make a class prediction $\hat{Y}$ for a new data input $\tilde{X} := (\tilde{X}_1,\ldots,\tilde{X}_d)$ according to

$$
\begin{align}
\hat{Y} &= \text{arg} \max_{y_k} p(Y=c_k | \tilde{X}_1,\ldots,\tilde{X}_d) \\
&= \text{arg} \max_{y_k}\frac{p(\tilde{X}_1,\ldots,\tilde{X}_d | Y=c_k)\, p(Y=y_k)}{\sum_{k=1}^K \, p(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)\, p(Y=c_k)}\\
&= \text{arg} \max_{c_k}\,  p(Y=c_k)\, \prod p(\tilde{X}_i| Y=c_k)\,
\end{align} \, ...... (2.4)$$

Since the denominator $p(X_1, X_2, \ldots , X_d)$ in our case is constant for a given input. We can use Maximum A Posteriori (MAP) estimation to estimate $p(Y=c_k)$ and $p(X_i|Y=c_k)$. The former is then the relative frequency of class in the training set.

The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of $p(X_i|Y=c_k)$. This is very important for the MAP estimation, for example if we assume class conditionals are univariate gaussians the parameters that need to be estimated are mean and standard deviations. The number of parameters depends on features and classes.

### 3. Connecting NB and Logistic Regression (Binary)

Instead of the generalized case above for NB classifier with $K$ classes, we simply consider 2 classes i.e. $Y$ is now boolean (0/1, True/False). Since Logistic Regression (LogReg) is a discriminative algorithm it starts from assuming a functional form for $P(Y|X)$ i.e. ---
$$\begin{equation} P(Y|X) = f(x) = \sigma(w^T\, x);\, \,  \\ \text{where},\,\,  \sigma(a) = \left(1 + \text{exp}(-a)\right)^{-1}\end{equation}\, ...... (3.0)$$

The parameters are estimated from the training data.

Let's write it in a bit more detail. The parametric form of LogReg for $Y$ as boolean can be written as --

$$\begin{align} P(Y=1|X) = \frac{1}{1 + \text{exp}\left(w_0 + \sum\limits_{i=0}^d w_i\, X_i \right)} ; P(Y=0|X) = \frac{\text{exp}\left(w_0 + \sum\limits_{i=0}^d w_i\, X_i \right)}{1 + \text{exp}\left(w_0 + \sum\limits_{i=0}^d w_i\, X_i \right)}  \end{align}\, ...... (3.1)$$

To assign Y=0 for a given X, we then impose a simple condition as below--

$$\begin{equation}\frac{P(Y=0|X)}{P(Y=1|X)} \gt 1 \, ; \text{exp}\left(w_0 + \sum\limits_{i=0}^d w_i\, X_i \right) \gt 1 \, ; w_0 + \sum\limits_{i=0}^d w_i\, X_i \gt 0\end{equation}$$

The last line is after taking logarithm of both sides.

_So the condition for classification is based on a linear expression._

We will use Gaussian Naive Bayes (GNB) Classifier and recover the form of $P(Y|X)$ and compare with the Logistic regression results.

------------------------------------
**GNB as Binary Classifier**
------------------------------------

Assumptions ---

1.   $Y$ has boolean form (i.e 0/1, True/False) and it's governed by a Bernoulli distribution.
2.   Since it's a GNB, for class conditionals $P(X_i|Y=c_k)$ we assume univariate Gaussians.
3.   For all $i$ and $j\neq i$,  $X_i, X_j$ are conditionally independent given $Y$.

Let's write $P(Y=1|X)$ using Bayes' Rule---

$$\begin{align} p(Y = 1| X) = \frac{p(Y=1)\, p(X| Y=1)\, }{ p(Y=1) \,  p(X| Y=1)\, + p(Y=0)\, p(X|Y=0)}\end{align}\, ...... (3.2)$$

$$\begin{align} p(Y = 1| X) = \frac{1}{ 1 + \frac{p(Y=0)\, p(X|Y=0)}{p(Y=1)\, p(X|Y=1)}}  =  \frac{1}{ 1 + \text{exp}\left( \text{ln}\, \frac{p(Y=0)\, p(X|Y=0)}{p(Y=1)\, p(X|Y=1)} \right)} \end{align}\, ...... (3.3)$$

Let's assume $P(Y=1|X) = \pi \implies P(Y=0|X) = 1-\pi$ and also use the conditional independence assumption ;

$$\begin{align} p(Y = 1| X) = \frac{1}{ 1 + \frac{p(Y=0)\, p(X|Y=0)}{p(Y=1)\, p(X|Y=1)}}  =  \frac{1}{ 1 + \text{exp}\left( \text{ln}\, \frac{1-\pi}{\pi\,} + \sum \limits_i \text{ln}\,\frac{ p(X_i|Y=0)}{ p(X_i|Y=1)} \right)} \end{align}\, ...... (3.4)$$

For class conditionals $P(X_i|Y=c_k)$ we assume univariate Gaussians with parameters $N(\mu_{ik}, \sigma _i)$, i.e., the standard deviations are independent of class $(Y)$.

$$\begin{align} \sum _i \text{ln}\,\frac{p(X_i|Y=0)}{p(X_i|Y=1)} &= \sum \limits_i \text{ln}\, \frac{\frac{1}{\sqrt{2\pi \sigma _i ^2}} \, \text{exp}\, \left(\frac{-\left(X_i - \mu _{i 0}\right)^2}{2\sigma_i^2} \right)}{\frac{1}{\sqrt{2\pi \sigma_i^2}} \, \text{exp}\, \left(\frac{-\left(X_i - \mu _{i 1}\right)^2}{2\sigma_i^2} \right)}\\ & = \sum _i \text{ln}\, \text{exp}\, \left(\frac{(X_i - \mu_{i1})^2 - (X_i - \mu_{i 0})^2}{2\sigma_i^2} \right)\\ & = \sum \limits_i \left(\frac{(X_i - \mu_{i1})^2 - (X_i - \mu_{i 0})^2}{2\sigma _i^2}\right)\\ & = \sum \limits_i \left( \frac{\mu_{i0} - \mu_{i1}}{\sigma_i^2}\, X_i \, + \, \frac{\mu_{i0}^2 - \mu_{i1}^2}{2\sigma_i^2}\, \right)  \, ...... (3.5) \end{align}$$

Let's put this back in equation 14.

$$\begin{align} p(Y = 1| X) = \frac{1}{ 1 + \frac{p(Y=0)\, p(X|Y=0)}{p(Y=1)\, p(X|Y=1)}}  =  \frac{1}{ 1 + \text{exp}\left( \text{ln}\, \frac{1-\pi}{\pi\,} + \sum \limits_i \left( \frac{\mu_{i0} - \mu_{i1}}{\sigma_i^2}\, X_i \, + \, \frac{\mu_{i1}^2 - \mu_{i0}^2}{2\sigma_i^2}\, \right)\right)} \end{align}\, ...... (3.6)$$

Let's compare directly with eq. 11,

$$P(Y=1|X) = \frac{1}{1 + \text{exp}\left(w_0 + \sum\limits_{i=0}^d w_i\, X_i \right)} $$

So we can write

$$w_0 = \text{ln}\, \frac{1-\pi}{\pi} + \sum_i \frac{\mu_{i1}^2 - \mu_{i0}^2}{2\sigma_i^2}; \, w_i = \frac{\mu_{i0} - \mu_{i1}}{2\sigma_i^2}\, ......\, (3.7)$$

So here we have derived a generative formulation for logistic regression.

```python
import math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
```

```python
from google.colab import drive
drive.mount('/content/drive/')
```

    Mounted at /content/drive/

```python
penguin = pd.read_csv('/content/drive/My Drive/Colab Notebooks/penguins.csv', sep=',')
penguin.head(5)
```

      species
      island
      bill_length_mm
      bill_depth_mm
      flipper_length_mm
      body_mass_g
      sex

      0
      Adelie
      Torgersen
      39.1
      18.7
      181.0
      3750.0
      male

      1
      Adelie
      Torgersen
      39.5
      17.4
      186.0
      3800.0
      female

      2
      Adelie
      Torgersen
      40.3
      18.0
      195.0
      3250.0
      female

      3
      Adelie
      Torgersen
      NaN
      NaN
      NaN
      NaN
      NaN

      4
      Adelie
      Torgersen
      36.7
      19.3
      193.0
      3450.0
      female

      <button class="colab-df-convert" onclick="convertToInteractive('df-7c275443-3ee7-46dd-bc6c-2869ac6d0bf3')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

```python
print (penguin.shape)

penguin_nonan_df = penguin.dropna(how='any', axis=0, inplace=False)
print (penguin_nonan_df.shape)

penguin_nonan_df.reset_index(inplace=True)
```

    (344, 7)
    (333, 7)

```python
fig = plt.figure(figsize=(7, 5))
sns_fgrid=sns.FacetGrid(penguin_nonan_df, hue="species", height=6).map(plt.scatter, "bill_length_mm", "bill_depth_mm").add_legend()
plt.xlabel('Bill Length (mm)', fontsize=12)
plt.ylabel('Bill Depth (mm)', fontsize=12)
#plt.savefig('/content/drive/My Drive/Colab Notebooks/Bill_L_D.png', dpi=200)
plt.show()
```

```python
#### select only bill_length and and bill_depth as features

penguin_nonan_selected_df = penguin_nonan_df[['species', 'bill_length_mm', 'bill_depth_mm']]
penguin_nonan_selected_df.head(3)
```

      species
      bill_length_mm
      bill_depth_mm

      0
      Adelie
      39.1
      18.7

      1
      Adelie
      39.5
      17.4

      2
      Adelie
      40.3
      18.0

      <button class="colab-df-convert" onclick="convertToInteractive('df-8a7c771a-be8a-4d03-ae3a-b1288c1fec36')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

```python
X=penguin_nonan_selected_df.drop(['species'], axis=1)
Y=penguin_nonan_selected_df['species']

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=30, stratify=Y)

print (X_train.shape, y_train.shape, X_test.shape)
```

    (266, 2) (266,) (67, 2)

```python
X_train = X_train.to_numpy()
X_test = X_test.to_numpy()

y_train = y_train.to_numpy()
y_test = y_test.to_numpy()
```

```python
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

print (tfp.__version__, tf.__version__)
```

    0.15.0 2.7.0

We need the class prior distribution.

Simply we count the number of unique classes and divide by the total number of samples to get the class prior distribution. Let's see below -- $$ P(Y=c_k)  = \frac{\sum \limits_i^N \, \delta(Y^{(i)} = y_k)}{N}\, ...... (4.0)$$

```python
def get_prior(y):
    """
    This function takes training labels as a numpy array y of shape (num_samples,) as an input.
    This function will build a Categorical Distribution object with empty batch shape
    and event shape, with the probability of each class given as above.
    Your function should return the Distribution object.
    """
    N = y.shape[0] # total number of samples
    unique, counts = np.unique(y, return_counts=True) # count the number of occurences of 0, 1, 2
    probs = counts/N
    print (probs)
    dist = tfd.Categorical(probs=probs)
    return dist
```

```python
prior = get_prior(y_train)

print (prior)
print ('check prior probs: ', prior.probs,)
```

    [0.43984962 0.20300752 0.35714286]
    tfp.distributions.Categorical("Categorical", batch_shape=[], event_shape=[], dtype=int32)
    check prior probs:  tf.Tensor([0.43984962 0.20300752 0.35714286], shape=(3,), dtype=float64)

For class conditionals univariate gaussians will be used as we've described before; $P(X_i | Y=c_k)$ we have 2 features thus $i=0, 1$ and 3 classes thus $k=0, 1, 2$.

$$
\begin{align}
P(X_i | Y=c_k) &= N(X_i | \mu_{ik}, \sigma_{ik})\\
&= \frac{1}{\sqrt{2\pi\sigma_{ik}^2}} \exp\left\{-\frac{1}{2} \left(\frac{x - \mu_{ik}}{\sigma_{ik}}\right)^2\right\}\, ......(4.1)
\end{align}
$$
with mean $\mu_{ik}$ and standard deviation $\sigma_{ik}$, 12 parameters in all. These parameters will be estimated by maximum likelihood estimation. The parameters can easily be caluclated for a Normal distribution (check [here](https://www.statlect.com/fundamentals-of-statistics/normal-distribution-maximum-likelihood)).

Mean is of the form $\mu  = \frac{1}{N}\sum \limits_{i=1}^N x_i$, and std can be written as $\sigma^2 = \frac{1}{N} \sum \limits_{i=1}^N (x_j - \mu)^2$.

In this case including the class info we can modify the expression as below (this can also be derived from the equation above $P(X_i|Y=c_k)$) --

$$\mu_{ik} = \frac{\sum_n X_i^{(n)} \delta(Y^{(n)}=c_k)}{\sum_n \delta(Y^{(n)}=c_k)} \, ...... (4.2)\\
\sigma^2_{ik} = \frac{\sum_n (X_i^{(n)} - \mu_{ik})^2 \delta(Y^{(n)}=c_k)}{\sum_n \delta(Y^{(n)}=c_k)}\, ...... (4.3)$$

```python
def class_conditionals_MLE(x, y):
    """
    x shape (samples, features) and  y shape (samples, ) .

    The batch shape of this distribution will be 3, it corresponds
    to the number of classes and the event shape corresponds to the number of features.
    This function will then return the Normal Distribution object.
    """
    unique = np.unique(y)
    idxs = [np.where(y==unique[0]), np.where(y==unique[1]), np.where(y==unique[2])]
    mu = []
    sigma2 = []
    for id in idxs:
        mu.append(list(np.mean(x[id], axis=0)) )
    for i, k in enumerate(idxs):
        sigma2.append(list(np.mean((x[k] - mu[i])**2, axis=0)))
    sigma = np.sqrt(sigma2)
    #print('mu and sigma: ', mu, '\n', sigma)
    dist = tfd.MultivariateNormalDiag(loc=mu, scale_diag=sigma)
    return dist, mu, sigma2
```

```python
class_conditionals, mu, sigma2 = class_conditionals_MLE(X_train, y_train)

print (class_conditionals)
print ('check mu and variance: ', '\n')

print ('mu: ', mu, )
print ('sigma2: ', sigma2, )
# batch shape : 3 classes, event shape : 2 features
```

    tfp.distributions.MultivariateNormalDiag("MultivariateNormalDiag", batch_shape=[3], event_shape=[2], dtype=float64)
    check mu and variance:

    mu:  [ListWrapper([39.017094017093996, 18.389743589743592]), ListWrapper([49.12592592592592, 18.479629629629624]), ListWrapper([48.063157894736854, 15.058947368421052])]
    sigma2:  [[7.205861640733441, 1.5171597633136102], [10.038216735253773, 1.1042146776406032], [9.476642659279785, 0.9687357340720222]]

#### What's Happening in Class Conditional Function

the upcoming 2 cells give a breakdown in detail of what's happening in the function above and why it can't be used later for binary classification.

```python
unique = np.unique(y_test)
idxs = [np.where(y_test==unique[0]), np.where(y_test==unique[1]), np.where(y_test==unique[2])]

print (unique)
print (idxs, len(idxs), '\n', type(idxs), idxs[0])

for i, k in enumerate(idxs):
  print (i, k)
for id in idxs:
  print (id)
```

    ['Adelie' 'Chinstrap' 'Gentoo']
    [(array([ 0,  3,  6,  8, 11, 12, 13, 14, 15, 17, 18, 22, 24, 25, 28, 29, 30,
           31, 38, 40, 42, 46, 49, 50, 51, 54, 57, 63, 64]),), (array([ 4,  5,  7, 10, 23, 26, 27, 32, 34, 39, 44, 53, 56, 58]),), (array([ 1,  2,  9, 16, 19, 20, 21, 33, 35, 36, 37, 41, 43, 45, 47, 48, 52,
           55, 59, 60, 61, 62, 65, 66]),)] 3
     <class 'list'> (array([ 0,  3,  6,  8, 11, 12, 13, 14, 15, 17, 18, 22, 24, 25, 28, 29, 30,
           31, 38, 40, 42, 46, 49, 50, 51, 54, 57, 63, 64]),)
    0 (array([ 0,  3,  6,  8, 11, 12, 13, 14, 15, 17, 18, 22, 24, 25, 28, 29, 30,
           31, 38, 40, 42, 46, 49, 50, 51, 54, 57, 63, 64]),)
    1 (array([ 4,  5,  7, 10, 23, 26, 27, 32, 34, 39, 44, 53, 56, 58]),)
    2 (array([ 1,  2,  9, 16, 19, 20, 21, 33, 35, 36, 37, 41, 43, 45, 47, 48, 52,
           55, 59, 60, 61, 62, 65, 66]),)
    (array([ 0,  3,  6,  8, 11, 12, 13, 14, 15, 17, 18, 22, 24, 25, 28, 29, 30,
           31, 38, 40, 42, 46, 49, 50, 51, 54, 57, 63, 64]),)
    (array([ 4,  5,  7, 10, 23, 26, 27, 32, 34, 39, 44, 53, 56, 58]),)
    (array([ 1,  2,  9, 16, 19, 20, 21, 33, 35, 36, 37, 41, 43, 45, 47, 48, 52,
           55, 59, 60, 61, 62, 65, 66]),)

```python
print (np.mean(X_train[[ 4,  5,  7, 10, 23, 26, 27, 32, 34, 39, 44, 53, 56, 58]],  axis=0))
mu2 = np.mean(X_train[[ 4,  5,  7, 10, 23, 26, 27, 32, 34, 39, 44, 53, 56, 58]],  axis=0)
print (X_train[[ 4,  5,  7, 10, 23, 26, 27, 32, 34, 39, 44, 53, 56, 58]])
print (X_train[4])

print (np.mean((X_train[[ 4,  5,  7, 10, 23, 26, 27, 32, 34, 39, 44, 53, 56, 58]] - mu2)**2, axis=0))
```

    [45.35714286 17.24285714]
    [[38.8 20. ]
     [40.3 18.5]
     [46.8 16.1]
     [59.6 17. ]
     [36.9 18.6]
     [58.  17.8]
     [50.7 15. ]
     [50.4 15.7]
     [38.9 18.8]
     [47.5 14. ]
     [45.7 17. ]
     [40.6 17.2]
     [37.3 20.5]
     [43.5 15.2]]
    [38.8 20. ]
    [49.73244898  3.42102041]

```python
print (class_conditionals.prob(X_train[1]))
#calculate the probability density for an example sample.
# this can also be thought of as likelihood
```

    tf.Tensor([2.28228213e-02 7.27188025e-06 5.03265210e-07], shape=(3,), dtype=float64)

Finally we make the predictions on the test set.
So we select sample by sample and use the info from prior probabilities and class conditionals.

$$
P(Y=c_k | \tilde{X}_1,\ldots,\tilde{X}_d) = \frac{P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=c_k)P(Y=c_k)}{\sum_{k=1}^K P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=c_k)}
\, ......\, (4.4)$$

The class prediction can then be taken as the class with the maximum probability:

$$
\hat{Y} = \text{arg}\, \max_{c_k} P\left(Y=c_k | \tilde{X}_1,\ldots,\tilde{X}_d \right) = \text{arg}\, \max_{c_k} \frac{P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=c_k)P(Y=c_k)}{\sum_{k=1}^K P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=c_k)}
\, ......\, (4.5)$$

```python

def predict_class(prior_dist, class_cond_dist, x):
    """
    We will use prior distribution (P(Y|C)), class-conditional distribution(P(X|Y)),
    and test data-set with shape (batch_shape, 2).
    """
    y = np.zeros((x.shape[0]), dtype=int)
    for i, train_point in enumerate(x):
        likelihood = tf.cast(class_cond_dist.prob(train_point), dtype=tf.float32)
        prior_prob = tf.cast(prior_dist.probs, dtype=tf.float32)
        numerator =  likelihood * prior_prob
        denominator = tf.reduce_sum(numerator)
        P = tf.math.divide(numerator, denominator)
        #print ('check posterior shape: ', P.shape)
        Y = tf.argmax(P) # exact similar to np.argmax [get the class]
        y[i] = int(Y)
    return y
```

```python
predictions = predict_class(prior, class_conditionals, X_test)

print (predictions, y_test)
```

    [0 2 2 0 0 2 0 0 0 2 1 0 0 0 0 0 2 0 0 2 2 2 0 1 0 0 1 1 0 0 0 0 1 2 2 2 2
     2 0 1 0 2 0 0 1 2 0 2 2 0 0 0 2 1 0 2 0 0 1 2 2 2 2 0 0 2 2] ['Adelie' 'Gentoo' 'Gentoo' 'Adelie' 'Chinstrap' 'Chinstrap' 'Adelie'
     'Chinstrap' 'Adelie' 'Gentoo' 'Chinstrap' 'Adelie' 'Adelie' 'Adelie'
     'Adelie' 'Adelie' 'Gentoo' 'Adelie' 'Adelie' 'Gentoo' 'Gentoo' 'Gentoo'
     'Adelie' 'Chinstrap' 'Adelie' 'Adelie' 'Chinstrap' 'Chinstrap' 'Adelie'
     'Adelie' 'Adelie' 'Adelie' 'Chinstrap' 'Gentoo' 'Chinstrap' 'Gentoo'
     'Gentoo' 'Gentoo' 'Adelie' 'Chinstrap' 'Adelie' 'Gentoo' 'Adelie'
     'Gentoo' 'Chinstrap' 'Gentoo' 'Adelie' 'Gentoo' 'Gentoo' 'Adelie'
     'Adelie' 'Adelie' 'Gentoo' 'Chinstrap' 'Adelie' 'Gentoo' 'Chinstrap'
     'Adelie' 'Chinstrap' 'Gentoo' 'Gentoo' 'Gentoo' 'Gentoo' 'Adelie'
     'Adelie' 'Gentoo' 'Gentoo']

```python
class_types = ['Adelie', 'Chinstrap', 'Gentoo']
class_types_dict = {'Adelie':0, 'Chinstrap':1, 'Gentoo':2}

labels = dict(zip(class_types_dict.values(), class_types_dict.keys()))
```

```python
from sklearn.metrics import accuracy_score

y_test_cat = np.vectorize(class_types_dict.get)(y_test)

print (y_test_cat)

accuracy = accuracy_score(y_test_cat, predictions)
print("Test accuracy: ", accuracy)
```

    [0 2 2 0 1 1 0 1 0 2 1 0 0 0 0 0 2 0 0 2 2 2 0 1 0 0 1 1 0 0 0 0 1 2 1 2 2
     2 0 1 0 2 0 2 1 2 0 2 2 0 0 0 2 1 0 2 1 0 1 2 2 2 2 0 0 2 2]
    Test accuracy:  0.9104477611940298

```python
def plot_data(x, y, labels, colors, set='Train'):
    fig = plt.figure(figsize=(8, 6))
    for c in np.unique(y):
        inx = np.where(y == c)
        plt.scatter(x[inx, 0], x[inx, 1], label=labels[c], c=colors[c])
    plt.title("%s set"%(set), fontsize=12)
    plt.xlabel("Bill Length (mm)", fontsize=12)
    plt.ylabel("Bill Depth (mm)", fontsize=12)
    plt.legend()

def get_meshgrid(x0_range, x1_range, num_points=100):
    x0 = np.linspace(x0_range[0], x0_range[1], num_points)
    x1 = np.linspace(x1_range[0], x1_range[1], num_points)
    return np.meshgrid(x0, x1)

def contour_plot(x0_range, x1_range, prob_fn, batch_shape, colours, levels=None, num_points=100):
    X0, X1 = get_meshgrid(x0_range, x1_range, num_points=num_points)
    Z = prob_fn(np.expand_dims(np.array([X0.ravel(), X1.ravel()]).T, 1))
    Z = np.array(Z).T.reshape(batch_shape, *X0.shape)
    for batch in np.arange(batch_shape):
        if levels:
            plt.contourf(X0, X1, Z[batch], alpha=0.2, colors=colours, levels=levels)
        else:
            plt.contour(X0, X1, Z[batch], colors=colours[batch], alpha=0.3)

labels = dict(zip(class_types_dict.values(), class_types_dict.keys()))
colors = ['Blue', 'LimeGreen', 'Pink']
```

```python
# # Plot the model's decision regions

y_train_cat = np.vectorize(class_types_dict.get)(y_train)

plt.figure(figsize=(10, 6))
plot_data(X_test, y_test_cat, labels, colors)
x0_min, x0_max = X_test[:, 0].min(), X_test[:, 0].max()
x1_min, x1_max = X_test[:, 1].min(), X_test[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max),
             lambda x: predict_class(prior, class_conditionals, x),
             1, colors, levels=[-0.5, 0.5, 1.5, 2.5],
             num_points=400)
plt.title("Test Set Decision Reg.", fontsize=10)
#plt.savefig('/content/drive/My Drive/Colab Notebooks/Decs_Reg_GNB.png', dpi=200)
plt.show()
```

```python
### try using sklearn

from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score

sklearn_GNB = GaussianNB()
sklearn_GNB.fit(X_train, y_train)

predictions_sklearn = sklearn_GNB.predict(X_test)

y_test_cat = np.vectorize(class_types_dict.get)(y_test)
predictions_sklearn_cat = np.vectorize(class_types_dict.get)(predictions_sklearn)

print (y_test_cat)
print (predictions_sklearn_cat)

accuracy_sklearn = accuracy_score(y_test_cat, predictions_sklearn_cat)
print("Test accuracy: ", accuracy_sklearn)

x0_min, x0_max = X_test[:, 0].min(), X_test[:, 0].max()
x1_min, x1_max = X_test[:, 1].min(), X_test[:, 1].max()

xx, yy = np.meshgrid(np.linspace(x0_min, x0_max, 100), np.linspace(x1_min, x1_max, 100))

Z = sklearn_GNB.predict_proba(np.c_[xx.ravel(), yy.ravel()])#[:, 0]

#Z_re = Z.reshape(3, *xx.shape)
Z_re = np.array(Z).T.reshape(3, *xx.shape)

print (Z.shape)
#print (Z_re.shape)
colors = ['Blue', 'LimeGreen', 'Pink']
levels=[-0.5, 0.5, 1.5, 2.5]
#for

plot_data(X_test, y_test_cat, labels, colors)
plt.contourf(xx, yy, Z_re[1], alpha=0.2,  levels=levels)
plt.contourf(xx, yy, Z_re[2], alpha=0.2, colors=colors, levels=levels)
plt.title('Test Set Decision Reg. Sklearn', fontsize=12)
#plt.savefig('/content/drive/My Drive/Colab Notebooks/sklearn_Penguin_GNB.png', dpi=200)
#plt.show()

print ('check the fit parameters: ', '\n')
print (sklearn_GNB.get_params(deep=True))
print ('variance:',  sklearn_GNB.var_)
print ('mean: ', sklearn_GNB.theta_)
```

#### Connecting Logistic Regression and Naive Bayes

As we discussed before, to connect Naive Bayes and logistic regression, we think of binary classification. Since we have 3 classes here in this Penguin dataset first we transform the problem as one vs rest classifier and then determine the logistic regression parameters.

Also on the derivation of GNB as a binary classifier, we have used same $\sigma _i$ for two different classes. So here we will be finding 6 parameters; 4 for $\mu _{ik}$ & 2 for $\sigma _i$.

The class conditional function we wrote before based on equation (20), and (21) depends on $\mu _{ik}$ and since this $\mu_{ik}$ is used in standard deviation formula, we can't use the same function as before because there's no way to get $\mu$ for a feature which is class independent.

So given the data samples we will learn the standard deviations by minimizing log likelihood given the data and labels. For distribution we will take MultivariateNormalDiag distribution.

```python
y_train_cat = np.vectorize(class_types_dict.get)(y_train)
y_train_cat_binary = y_train_cat

y_train_cat_binary[np.where(y_train_cat_binary == 2)] = 1 # gentoo = chinstrap

y_test_cat = np.vectorize(class_types_dict.get)(y_test)
y_test_cat_binary = np.array(y_test_cat)

y_test_cat_binary[np.where(y_test_cat_binary == 2)] = 1

print ('check shapes: ', y_train_cat_binary.shape, y_test_cat_binary.shape, y_train_cat_binary.dtype, '\n', y_train_cat.shape, y_train_cat)
```

    check shapes:  (266,) (67,) int64
     (266,) [0 0 0 1 0 0 0 1 0 0 1 1 0 1 1 1 0 0 0 1 1 1 1 0 0 0 1 1 1 1 0 1 1 0 0 0 0
     0 0 1 0 0 1 1 1 0 0 1 0 1 1 0 1 0 1 0 0 1 1 1 0 0 1 1 1 1 0 1 1 1 0 0 1 1
     1 1 1 1 0 0 1 0 0 0 1 1 0 0 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 0 0 1 0 1 1 1 1
     0 1 0 0 1 0 1 1 1 1 1 1 0 1 0 1 1 0 1 1 1 1 1 1 1 0 0 1 0 0 0 1 1 1 1 1 1
     1 1 1 1 1 1 0 1 1 0 1 1 0 0 0 0 0 1 1 1 0 1 1 1 1 0 1 0 1 0 1 1 0 0 0 1 0
     1 0 0 1 1 1 0 0 1 0 0 0 1 0 1 1 1 0 0 1 0 1 0 0 0 0 1 0 1 1 0 1 1 0 1 1 1
     1 1 0 1 0 1 0 0 0 1 0 1 1 0 1 1 1 1 0 0 0 1 0 0 1 0 1 0 0 0 1 1 0 1 1 0 0
     1 0 1 1 1 0 1]

```python
# Plot the training data

labels_binary = {0: 'Adelie', 1: 'Gentoo/Chinstrap'}
colors_binary = ['blue', 'red',]

plot_data(X_train, y_train_cat_binary, labels_binary, colors_binary)
#plt.savefig('/content/drive/My Drive/Colab Notebooks/sklearn_Penguin_GNB_2class.png', dpi=200)
```

```python
#y_train_cat_binary = y_train_cat_binary.astype('float32')
#y_test_cat_binary = y_test_cat_binary.astype('float32')

prior_binary = get_prior(y_train_cat_binary)
print (prior_binary.probs, print (type(prior_binary)))
```

    [0.43984962 0.56015038]
    <class 'tensorflow_probability.python.distributions.categorical.Categorical'>
    tf.Tensor([0.43984962 0.56015038], shape=(2,), dtype=float64) None

```python
def learn_stdevs(x, y, scales, optimiser, epochs):
    """
    Input: data points, targets, standard dev. variable, optimiser (Adam) and epochs.
    Trainable variables will be the standard dev. (scales) in a custom training loop.
    For log likelihood we will use MultivariateNormalDiag distribution object, this will be learned over epochs.
    """
    def compute_mu(x_train, y_train): # this is exactly same as used before in class conditionals
        mu = []
        for value in np.unique(y):
            idx = np.where(y==value)
            mu.append(list(np.mean(x[idx], axis=0)) )
        return np.array(mu).astype(np.float32)

    @tf.function # converts the regular def to a tensorflow graph function
    def nll(x_train, y_train):#this can later be used under GradientTape
        log_probs = dist.log_prob(x_train)
        y_train = tf.one_hot(y_train, len(tf.unique(y_train))) # turn to categorical distribution
        return -tf.reduce_mean(log_probs * y_train) # distrbution log prob * y_train

    def compute_loss_grads(x_train, y_train, dist):
        with tf.GradientTape() as g: # performs the differentiation for backpropagation
            g.watch(dist.trainable_variables)
            loss = nll(x_train, y_train)
            grads = g.gradient(loss, dist.trainable_variables)
        return loss, grads

    nlls, scales_arr = [], []
    print('x shape before: ', x.shape)
    mu = compute_mu(x, y)
    print('loc shape: ', mu.shape)
    x = np.expand_dims(x, axis=1).astype(np.float32)
    print('x shape after: ', x.shape)
    dist = tfd.MultivariateNormalDiag(loc=mu, scale_diag=scales) # define the multivariate normal dist., with scales as trainable params.
    for epoch in range(epochs):
        loss, grads = compute_loss_grads(x, y, dist)
        optimiser.apply_gradients(zip(grads, dist.trainable_variables))
        if epoch%20==0:
          print("Epoch \t Loss: ", epoch, loss)
        nlls.append(loss)
        scales_arr.append(scales.numpy()) # transforms the input tensor to numpy array via scales.numpy()
    nlls, scales_arr = np.array(nlls), np.array(scales_arr)
    return nlls, scales_arr, dist
```

```python
scales = tf.Variable([5., 5.]) # this is trainable params.
print (scales)
opt = tf.keras.optimizers.Adam(learning_rate=1e-2)
epochs = 600
```

```python
nlls, scales_arr, class_conditionals_binary = learn_stdevs(X_train, y_train_cat_binary, scales, opt,
                                                           epochs)
```

    Epoch 	 Loss:  0 tf.Tensor(2.6432767, shape=(), dtype=float32)
    Epoch 	 Loss:  20 tf.Tensor(2.612201, shape=(), dtype=float32)
    Epoch 	 Loss:  40 tf.Tensor(2.5805533, shape=(), dtype=float32)
    Epoch 	 Loss:  60 tf.Tensor(2.5483735, shape=(), dtype=float32)
    Epoch 	 Loss:  80 tf.Tensor(2.5157988, shape=(), dtype=float32)
    Epoch 	 Loss:  100 tf.Tensor(2.4830365, shape=(), dtype=float32)
    Epoch 	 Loss:  120 tf.Tensor(2.450353, shape=(), dtype=float32)
    Epoch 	 Loss:  140 tf.Tensor(2.4180593, shape=(), dtype=float32)
    Epoch 	 Loss:  160 tf.Tensor(2.3864856, shape=(), dtype=float32)
    Epoch 	 Loss:  180 tf.Tensor(2.355945, shape=(), dtype=float32)
    Epoch 	 Loss:  200 tf.Tensor(2.3267198, shape=(), dtype=float32)
    Epoch 	 Loss:  220 tf.Tensor(2.299125, shape=(), dtype=float32)
    Epoch 	 Loss:  240 tf.Tensor(2.2736912, shape=(), dtype=float32)
    Epoch 	 Loss:  260 tf.Tensor(2.2514012, shape=(), dtype=float32)
    Epoch 	 Loss:  280 tf.Tensor(2.2337108, shape=(), dtype=float32)
    Epoch 	 Loss:  300 tf.Tensor(2.2219605, shape=(), dtype=float32)
    Epoch 	 Loss:  320 tf.Tensor(2.2161384, shape=(), dtype=float32)
    Epoch 	 Loss:  340 tf.Tensor(2.2142801, shape=(), dtype=float32)
    Epoch 	 Loss:  360 tf.Tensor(2.2139611, shape=(), dtype=float32)
    Epoch 	 Loss:  380 tf.Tensor(2.2139401, shape=(), dtype=float32)
    Epoch 	 Loss:  400 tf.Tensor(2.2139406, shape=(), dtype=float32)
    Epoch 	 Loss:  420 tf.Tensor(2.2139404, shape=(), dtype=float32)
    Epoch 	 Loss:  440 tf.Tensor(2.2139404, shape=(), dtype=float32)
    Epoch 	 Loss:  460 tf.Tensor(2.2139401, shape=(), dtype=float32)
    Epoch 	 Loss:  480 tf.Tensor(2.2139401, shape=(), dtype=float32)
    Epoch 	 Loss:  500 tf.Tensor(2.2139401, shape=(), dtype=float32)
    Epoch 	 Loss:  520 tf.Tensor(2.2139401, shape=(), dtype=float32)
    Epoch 	 Loss:  540 tf.Tensor(2.2139401, shape=(), dtype=float32)
    Epoch 	 Loss:  560 tf.Tensor(2.2139401, shape=(), dtype=float32)
    Epoch 	 Loss:  580 tf.Tensor(2.2139401, shape=(), dtype=float32)

```python
print (class_conditionals_binary.parameters)
```

    {'loc': array([[39.017094, 18.389744],
           [48.448322, 16.298658]], dtype=float32), 'scale_diag': <tf.Variable 'Variable:0' shape=(2,) dtype=float32, numpy=array([2.9560146, 1.6589097], dtype=float32)>, 'scale_identity_multiplier': None, 'validate_args': False, 'allow_nan_stats': True, 'experimental_use_kahan_sum': False, 'name': 'MultivariateNormalDiag'}

```python
print("Class conditional means:")
print(class_conditionals_binary.loc.numpy())
print("\nClass conditional standard deviations:")
print(class_conditionals_binary.stddev().numpy())
```

    Class conditional means:
    [[39.017094 18.389744]
     [48.448322 16.298658]]

    Class conditional standard deviations:
    [[2.9560146 1.6589097]
     [2.9560146 1.6589097]]

```python
print (class_conditionals_binary.covariance().numpy())
cov_mat = class_conditionals_binary.covariance().numpy()
print (cov_mat[0], '\n', cov_mat[1])
```

    [[[8.738023  0.       ]
      [0.        2.7519813]]

     [[8.738023  0.       ]
      [0.        2.7519813]]]
    [[8.738023  0.       ]
     [0.        2.7519813]]
     [[8.738023  0.       ]
     [0.        2.7519813]]

Here we see that there are only 2 different numbers for standard deviations. That's what we expected...

```python
fig = plt.figure(figsize=(6, 4))
for k in [0, 1]:
    plt.plot(scales_arr[:, k], color=colors_binary[k], label=labels_binary[k])
plt.title("Stand Dev. vs Epoch")
plt.xlabel("Epochs")
plt.ylabel("Standard Deviation")
plt.legend()
plt.show()
```

```python
plt.figure(figsize=(10, 6))
plot_data(X_test, y_test_cat_binary, labels_binary, colors_binary)
x0_min, x0_max = X_test[:, 0].min(), X_test[:, 0].max()
x1_min, x1_max = X_test[:, 1].min(), X_test[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max), class_conditionals_binary.prob, 2, colors_binary)
plt.title("Training set with class-conditional density contours")
plt.show()
```

```python
# # Plot the model's decision regions

plt.figure(figsize=(10, 6))
plot_data(X_test, y_test_cat_binary, labels_binary, colors_binary)
x0_min, x0_max = X_test[:, 0].min(), X_test[:, 0].max()
x1_min, x1_max = X_test[:, 1].min(), X_test[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max),
             lambda x: predict_class(prior_binary, class_conditionals_binary, x),
             1, colors_binary, levels=[-0.5, 0.5, 1.5],
             num_points=300)
plt.title("Test set with Decision Reg.", fontsize=11)
plt.savefig('/content/drive/My Drive/Colab Notebooks/sklearn_Penguin_GNB_2class_Decs_Reg.png')
plt.show()
```

#### Deriving Parameters for Logistic Regression
**Using Binary Gaussian Naive Bayes Classifier**

We have already derived the parameters before, here we rewrite them in  more compact manner.

$$w_0 = \text{ln}\, \frac{1-\pi}{\pi} + \sum_i \frac{\mu_{i1}^2 - \mu_{i0}^2}{2\sigma_i^2}; \, w_i = \frac{\mu_{i0} - \mu_{i1}}{2\sigma_i^2}\, ......\, (5.1)$$

For Multivariate Normal distribution we usually express it in terms of mean vector $\mu$, and covariance matrix $\Sigma$.

$$w_0 = \text{log}\, \frac{P(Y=Y_0)}{P(Y=Y_1)} + \frac{1}{2}\mu_1^T\, \Sigma^{-1}\, \mu_1 - \frac{1}{2}\mu_0^T\, \Sigma^{-1}\, \mu_0 ; \, w = \frac{\mu_{0} - \mu_{1}}{\Sigma}\, ......\, (5.2)$$

```python

def get_logistic_regression_params(prior, class_conditionals):
    """
    We will use the prior distribution and class-conditional distribution as inputs.
    then using the formulas above we will calculate the weight and bias term,
    function returns a 2-tuple of numpy arrays of shapes (2,) and () respectively.
    """
    cov = class_conditionals.covariance().numpy() #Sigma
    cov = cov[1] # covariance matrix is of shape [2, 2, 2], so we choose one of the entries in first dim. as cov[0]==cov[1]
    cov_inv = np.linalg.inv(cov) #Sigma inverse
    print('covariance matrix: ', cov)
    #print(cov_inv, end='\n\n')

    mu = class_conditionals.loc.numpy()
    print('mean vector: ', mu, end='\n\n')

    log_p = np.log(prior.prob(0) / prior.prob(1))
    print('ratio of priors: ', log_p, end='\n\n')

    weight = cov_inv @ (mu[0] - mu[1])
    print('weight: ', weight, end='\n\n')

    bias = -0.5 * (mu[0].T @ cov_inv @ mu[0]) + 0.5 * (mu[1].T @ cov_inv @ mu[1]) + log_p
    print('bias: ', bias)

    return weight, bias
```

```python
weight, bias = get_logistic_regression_params(prior_binary, class_conditionals_binary)
```

    covariance matrix:  [[0.3208845  0.        ]
     [0.         0.10949251]]
    mean vector:  [[5.0324326 3.445946 ]
     [6.2445784 2.851807 ]]

    ratio of priors:  -0.8079226951523734

    weight:  [-3.7775145  5.426297 ]

    bias:  3.4048687477187203

```python
### use sklearn logistic reg.

from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import SGDClassifier

sklearn_LogReg = LogisticRegression(penalty='l2', class_weight='none', max_iter=50, solver='lbfgs', C=1, intercept_scaling=1) # multi_class: ovr for binary problem
#sklearn_LogReg = SGDClassifier(loss='log', max_iter=2000, tol=1e-5, class_weight={1:0.43984962, 0:0.56015038})

sklearn_LogReg.fit(X_train, y_train_cat_binary)

predictions_sklearn = sklearn_LogReg.predict(X_test)

#y_test_cat = np.vectorize(class_types_dict.get)(y_test)
class_types_dict_binary = {'Adelie':0, 'Chinstrap/Gentoo':1}

#predictions_sklearn_cat_binary = np.vectorize(class_types_dict_binary.get)(predictions_sklearn)

print (y_test_cat_binary)
print (predictions_sklearn)

accuracy_sklearn = accuracy_score(y_test_cat_binary, predictions_sklearn)
print("Test accuracy Log Reg: ", accuracy_sklearn)

x0_min, x0_max = X_test[:, 0].min(), X_test[:, 0].max()
x1_min, x1_max = X_test[:, 1].min(), X_test[:, 1].max()

xx, yy = np.meshgrid(np.linspace(x0_min, x0_max, 100), np.linspace(x1_min, x1_max, 100))

Z = sklearn_LogReg.predict_proba(np.c_[xx.ravel(), yy.ravel()])#[:, 0]

#Z_re = Z.reshape(3, *xx.shape)
Z_re = np.array(Z).T.reshape(2, *xx.shape) # 2 features

print (Z.shape)
#print (Z_re.shape)
colors = ['Blue', 'LimeGreen', 'Pink']
levels=[-0.5, 0.5, 1.5]
#for

plot_data(X_test, y_test_cat_binary, labels_binary, colors_binary)
plt.contourf(xx, yy, Z_re[0], alpha=0.2,  levels=levels)
#plt.contourf(xx, yy, Z_re[2], alpha=0.2, colors=colors, levels=levels)
plt.title('Test Set Decision Reg. Sklearn', fontsize=12)
plt.show()

print ('check the fit parameters: ', '\n')
print (sklearn_LogReg.get_params(deep=True))
print ('weights:',  sklearn_LogReg.coef_)
print ('bias: ', sklearn_LogReg.intercept_)
```

### Check with Iris Dataset

```python
from sklearn import datasets, model_selection
iris = datasets.load_iris()
data = iris.data[:, :2]
targets = iris.target

x_train, x_test, y_train, y_test = model_selection.train_test_split(data, targets, test_size=0.2)

print ('train feature shape: ', x_train.shape, x_train[0], x_train[10])
print ('train label shape: ', y_train.shape, y_train[0], y_train[10])
```

    train feature shape:  (120, 2) [6.7 2.5] [6.7 3. ]
    train label shape:  (120,) 2 2

```python
prior = get_prior(y_train)

print (prior)
print (prior.probs)

class_conditionals, mu, sigma2 = class_conditionals_MLE(x_train, y_train)
print ('\n')
print ('mu: ', mu, )
print ('sigma2: ', sigma2)
# batch shape : 3 classes, event shape : 2 features

print (class_conditionals)
```

    [0.30833333 0.34166667 0.35      ]
    tfp.distributions.Categorical("Categorical", batch_shape=[], event_shape=[], dtype=int32)
    tf.Tensor([0.30833333 0.34166667 0.35      ], shape=(3,), dtype=float64)

    mu:  [ListWrapper([5.032432432432434, 3.4459459459459465]), ListWrapper([5.936585365853658, 2.760975609756098]), ListWrapper([6.545238095238096, 2.9404761904761902])]
    sigma2:  [[0.13462381300219142, 0.1235646457268079], [0.26475907198096366, 0.10433075550267697], [0.35676303854875296, 0.08621882086167802]]
    tfp.distributions.MultivariateNormalDiag("MultivariateNormalDiag", batch_shape=[3], event_shape=[2], dtype=float64)

```python
# Get the class predictions

predictions = predict_class(prior, class_conditionals, x_test)

print (predictions)

from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, predictions)
print("Test accuracy: {:.4f}".format(accuracy))
```

    [0 1 0 0 1 1 0 2 2 0 2 1 1 2 0 1 1 0 2 0 0 1 2 0 2 1 0 2 1 0]
    Test accuracy: 0.8333

```python
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score

sklearn_GNB = GaussianNB()
sklearn_GNB.fit(x_train, y_train)

predictions_sklearn = sklearn_GNB.predict(x_test)

#y_test_cat = np.vectorize(class_types_dict.get)(y_test)
#predictions_sklearn_cat = np.vectorize(class_types_dict.get)(predictions_sklearn)

#print (y_test_cat)
print (predictions_sklearn)

accuracy_sklearn = accuracy_score(y_test, predictions_sklearn)
print("Test accuracy sklearn: ", accuracy_sklearn)
```

    [0 1 0 0 1 1 0 2 2 0 2 1 1 2 0 1 1 0 2 0 0 1 2 0 2 1 0 2 1 0]
    Test accuracy sklearn:  0.8333333333333334

```python
print ('check the fit parameters: ', '\n')
print (sklearn_GNB.get_params(deep=True))
print ('variance:',  sklearn_GNB.var_)
print ('mean: ', sklearn_GNB.theta_)
```

    check the fit parameters:

    {'priors': None, 'var_smoothing': 1e-09}
    variance: [[0.13462381 0.12356465]
     [0.26475907 0.10433076]
     [0.35676304 0.08621882]]
    mean:  [[5.03243243 3.44594595]
     [5.93658537 2.76097561]
     [6.5452381  2.94047619]]

```python
y_train_binary = np.array(y_train)
y_train_binary[np.where(y_train_binary == 2)] = 1

y_test_binary = np.array(y_test)
y_test_binary[np.where(y_test_binary == 2)] = 1

print ('check shapes: ', y_train_binary.shape, y_test_binary.shape)
print ('y test binary: ', y_test_binary)
```

    check shapes:  (120,) (30,)
    y test binary:  [0 1 0 0 1 1 0 1 1 0 1 1 0 1 0 1 1 0 1 0 0 1 1 0 1 1 0 1 1 0]

```python
prior_binary = get_prior(y_train_binary)
print (prior_binary.probs)
```

    [0.30833333 0.69166667]
    tf.Tensor([0.30833333 0.69166667], shape=(2,), dtype=float64)

```python
scales = tf.Variable([1., 1.] )
print (scales)
opt = tf.keras.optimizers.Adam(learning_rate=0.01)
epochs = 300

nlls, scales_arr, class_conditionals_binary = learn_stdevs(x_train, y_train_binary, scales, opt, epochs)
```

    <tf.Variable 'Variable:0' shape=(2,) dtype=float32, numpy=array([1., 1.], dtype=float32)>
    x shape before:  (120, 2)
    loc shape:  (2, 2)
    x shape after:  (120, 1, 2)
    Epoch 	 Loss:  0 tf.Tensor(1.0265328, shape=(), dtype=float32)
    Epoch 	 Loss:  20 tf.Tensor(0.86319447, shape=(), dtype=float32)
    Epoch 	 Loss:  40 tf.Tensor(0.702211, shape=(), dtype=float32)
    Epoch 	 Loss:  60 tf.Tensor(0.591175, shape=(), dtype=float32)
    Epoch 	 Loss:  80 tf.Tensor(0.58370197, shape=(), dtype=float32)
    Epoch 	 Loss:  100 tf.Tensor(0.58212346, shape=(), dtype=float32)
    Epoch 	 Loss:  120 tf.Tensor(0.5818395, shape=(), dtype=float32)
    Epoch 	 Loss:  140 tf.Tensor(0.58180135, shape=(), dtype=float32)
    Epoch 	 Loss:  160 tf.Tensor(0.5817963, shape=(), dtype=float32)
    Epoch 	 Loss:  180 tf.Tensor(0.5817957, shape=(), dtype=float32)
    Epoch 	 Loss:  200 tf.Tensor(0.5817955, shape=(), dtype=float32)
    Epoch 	 Loss:  220 tf.Tensor(0.5817956, shape=(), dtype=float32)
    Epoch 	 Loss:  240 tf.Tensor(0.58179563, shape=(), dtype=float32)
    Epoch 	 Loss:  260 tf.Tensor(0.5817956, shape=(), dtype=float32)
    Epoch 	 Loss:  280 tf.Tensor(0.5817956, shape=(), dtype=float32)

```python
# View the distribution parameters

print("Class conditional means:")
print(class_conditionals_binary.loc.numpy())
print("\nClass conditional standard deviations:")
print(class_conditionals_binary.stddev().numpy())
print(np.shape(class_conditionals_binary.stddev().numpy()))

print(class_conditionals_binary.covariance().numpy())
print('square of stddev: ', np.square(class_conditionals_binary.stddev().numpy()))
print('\n')
print(np.shape(class_conditionals_binary.covariance().numpy()))
print(class_conditionals_binary.parameters)
```

    Class conditional means:
    [[5.0324326 3.445946 ]
     [6.2445784 2.851807 ]]

    Class conditional standard deviations:
    [[0.5664667  0.33089653]
     [0.5664667  0.33089653]]
    (2, 2)
    [[[0.3208845  0.        ]
      [0.         0.10949251]]

     [[0.3208845  0.        ]
      [0.         0.10949251]]]
    square of stddev:  [[0.3208845  0.10949251]
     [0.3208845  0.10949251]]

    (2, 2, 2)
    {'loc': array([[5.0324326, 3.445946 ],
           [6.2445784, 2.851807 ]], dtype=float32), 'scale_diag': <tf.Variable 'Variable:0' shape=(2,) dtype=float32, numpy=array([0.5664667 , 0.33089653], dtype=float32)>, 'scale_identity_multiplier': None, 'validate_args': False, 'allow_nan_stats': True, 'experimental_use_kahan_sum': False, 'name': 'MultivariateNormalDiag'}

```python
w, w0 = get_logistic_regression_params(prior_binary, class_conditionals_binary)
print ('weight and bias: ', w, w0)
```

    covariance matrix:  [[0.3208845  0.        ]
     [0.         0.10949251]]
    mean vector:  [[5.0324326 3.445946 ]
     [6.2445784 2.851807 ]]

    ratio of priors:  -0.8079226951523734

    weight:  [-3.7775145  5.426297 ]

    bias:  3.4048687477187203
    weight and bias:  [-3.7775145  5.426297 ] 3.4048687477187203

```python
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import SGDClassifier

sklearn_LogReg = LogisticRegression(class_weight='balanced', max_iter=5000, solver='sag', C=1, intercept_scaling=1) # multi_class: ovr for binary problem
#sklearn_LogReg = SGDClassifier(loss='log', max_iter=2000, tol=1e-5, class_weight={1:0.43984962, 0:0.56015038})

sklearn_LogReg.fit(x_train, y_train_binary)

predictions_sklearn = sklearn_LogReg.predict(x_test)

print ('check the fit parameters: ', '\n')
print (sklearn_LogReg.get_params(deep=True))
print ('weights:',  sklearn_LogReg.coef_)
print ('bias: ', sklearn_LogReg.intercept_)
```

    check the fit parameters:

    {'C': 1, 'class_weight': 'balanced', 'dual': False, 'fit_intercept': True, 'intercept_scaling': 1, 'l1_ratio': None, 'max_iter': 5000, 'multi_class': 'auto', 'n_jobs': None, 'penalty': 'l2', 'random_state': None, 'solver': 'sag', 'tol': 0.0001, 'verbose': 0, 'warm_start': False}
    weights: [[ 3.03190154 -3.19091129]]
    bias:  [-6.75190939]

### Define a Custom Logistic Regression Classifier

```python
def sigmoid(z):
  return 1.0/(1 + np.exp(-z))
def loss(y, y_hat):
  loss = -np.mean(y*(np.log(y_hat)) - (1-y)*np.log(1-y_hat))
  return loss

def gradients(X, y, y_hat):

  # X --> Input.
  # y --> true/target value.
  # y_hat --> hypothesis/predictions.
  # w --> weights (parameter).
  # b --> bias (parameter).

  # m-> number of training examples.
  m = X.shape[0]

  # Gradient of loss w.r.t weights.
  dw = (1/m)*np.dot(X.T, (y_hat - y))

  # Gradient of loss w.r.t bias.
  db = (1/m)*np.sum((y_hat - y))

  return dw, db

def train(X, y, bs, epochs, lr):

    # X --> Input.
    # y --> true/target value.
    # bs --> Batch Size.
    # epochs --> Number of iterations.
    # lr --> Learning rate.

    # m-> number of training examples
    # n-> number of features
    m, n = X.shape

    # Initializing weights and bias to zeros.
    w = np.zeros((n,1))
    b = 0

    # Reshaping y.
    y = y.reshape(m,1)

    # Normalizing the inputs.
    x = X

    # Empty list to store losses.
    losses = []

    # Training loop.
    for epoch in range(epochs):
        for i in range((m-1)//bs + 1):

            # Defining batches. SGD.
            start_i = i*bs
            end_i = start_i + bs
            xb = X[start_i:end_i]
            yb = y[start_i:end_i]

            # Calculating hypothesis/prediction.
            y_hat = sigmoid(np.dot(xb, w) + b)

            # Getting the gradients of loss w.r.t parameters.
            dw, db = gradients(xb, yb, y_hat)

            # Updating the parameters.
            w -= lr*dw
            b -= lr*db

        # Calculating loss and appending it in the list.
        l = loss(y, sigmoid(np.dot(X, w) + b))
        losses.append(l)

    # returning weights, bias and losses(List).
    return w, b, losses

def predict(X):

    # X --> Input.

    # Normalizing the inputs.
    x = X

    # Calculating presictions/y_hat.
    preds = sigmoid(np.dot(X, w) + b)

    # Empty List to store predictions.
    pred_class = []    # if y_hat >= 0.5 --> round up to 1
    # if y_hat < 0.5 --> round up to 1
    pred_class = [1 if i > 0.5 else 0 for i in preds]

    return np.array(pred_class)

w, b, l = train(x_train, y_train_binary, bs=1, epochs=2000, lr=5e-3)
print (w, b)
```

    [[  6.54484322]
     [-10.61926577]] -1.9100633863956042

# !!!!!! End !!!!!!

```python

```


# A Basis for Grayscale Images

Paolo Prandoni, LCAV, EPFL

The concept of "basis" for a vector space is one of the most profound and fruitful ideas in linear algebra. Basis vectors are the "building blocks" that  all other vectors are built from; by "disassembling" a vector into a linear combination of basis elements we are able to "look inside" the vector and highlight many of its relevant features.

A vector space has an infinite number of bases, but not all bases are created equal; only a carefully crafted basis and, in particular, an orthogonal basis, can show potential in practical applications. In this notebook we will study how a special basis for grayscale images, called the **Haar basis**, can be used to build an efficient compression algorithm that is also robust to errors when used to transmit visual data.

```python
# usual pyton bookkeeping...
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import IPython
from IPython.display import Image
import math
```

```python
plt.rcParams["figure.figsize"] = (14,4)
# ensure all images will be grayscale
plt.gray()
```

```python
def multishow(*images):
    fig, axes = plt.subplots(nrows=1, ncols=len(images))
    for i, s in enumerate(images):
        axes[i].matshow(s);
    plt.show()
```

## 1. The space of MxN matrices

A digital grayscale image of size $M\times N$ consists of $MN$ real values encoding the intensity level of each pixel: the image data can therefore be represented by a real-valued matrix of size $M\times N$.

The set of all matrices of size $M\times N$ forms a vector space if we define addition, scalar multiplication and inner product as:

\begin{align}
\mathbf{A} + \mathbf{B} &=
    \left[
        \begin{array}{ccc}
            a_{0,0} & \dots & a_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    +
    \left[
        \begin{array}{ccc}
            b_{0,0} & \dots & b_{0,N-1} \\
            \vdots & & \vdots \\
            b_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    =
    \left[
        \begin{array}{ccc}
            a_{0,0}+b_{0,0} & \dots & a_{0,N-1}+b_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0}+b_{M-1,0} & \dots & a_{M-1,N-1}+b_{M-1,N-1}
        \end{array}
    \right]
    \\ \\
\beta\mathbf{A} &=
    \left[
        \begin{array}{ccc}
            \beta a_{0,0} & \dots & \beta a_{0,N-1} \\
            \vdots & & \vdots \\
            \beta a_{M-1,0} & \dots & \beta a_{M-1,N-1}
        \end{array}
    \right]
    \\ \\
    \langle \mathbf{A}, \mathbf{B} \rangle &= \sum_{m=0}^{M-1} \sum_{n=0}^{N-1} a_{m,n} b_{m, n}
\end{align}

In particualr, we have omitted the conjugation operator in the inner product since we will only deal with real-valued matrices. The inner product allows us to define orthogonality between images and this is rather useful since we're going to explore a couple of bases for this space.

Please note that the space of real-valued $M\times N$ matrices is isomorphic to $\mathbb{R}^{MN}$, that is, the space of real-valued Euclidean vectors of size $MN$, since we can always "unroll" a matrix into a vector. Assume we proceed column by column; then the matrix becomes the vector

$$
    \mathbf{a} = \mathbf{A}[:] = [
        \begin{array}{ccccccc}
            a_{0,0} & \dots & a_{M-1,0} & a_{0,1} & \dots & a_{M-1,1} & \ldots & a_{0, N-1} & \dots & a_{M-1,N-1}
        \end{array}]^T
$$

It is easy to see that the vector space operations defined for matrices are consistent with the standard operations in  $\mathbb{R}^{MN}$. Although the matrix and vector forms represent exactly the same data, the matrix form allows us to display the data in the familiar shape of an image.

## 2. Manipulating and displaying images

In numpy, the equivalence between matrices and images is fully acknowledged. As an example we can create a checkerboard pattern of any size with the following function:

```python
# let's create a checkerboard pattern
SIZE = 4
img = np.zeros((SIZE, SIZE))
for n in range(0, SIZE):
    for m in range(0, SIZE):
        if (n & 0x1) ^ (m & 0x1):
            img[n, m] = 1
```

and display the matrix with `matshow()`; note that the plotting routine automatically rescales the values in the matrix so that the smallest number is mapped to black and the largest to white.

```python
plt.matshow(img);
```

Conveniently, using IPython, we can read images from disk in any given format and convert them to numpy arrays; let's load and display for instance a JPEG image:

```python
img = np.array(plt.imread('cameraman.jpg'), dtype=int)
plt.matshow(img);
```

The image is a $64\times 64$ low-resolution version of the famous "cameraman" test picture. Out of curiosity, we can look at the first column of this image, which is is a $64×1$ vector:

```python
img[:,0]
```

The values are integers between zero and 255, meaning that each pixel is encoded over 8 bits (or 256 gray levels), with zero representing black and 255 representing white.

## 3. The canonical basis for images

The canonical basis for any matrix space $\mathbb{R}^{M\times N}$ is the set of "delta" matrices where only one element is equal to one while all the others are 0. Let's call them $\mathbf{E}_n$ with $0 \leq n < MN$. Here is a function to create the canonical basis vector given its index:

```python
def canonical(n, M=5, N=10):
    e = np.zeros((M, N))
    e[(n % M), int(n / M)] = 1
    return e
```

Here are some basis vectors: look for the position of white pixel, which differentiates them and note that we enumerate pixels column-wise:

```python
multishow(canonical(0), canonical(1), canonical(49));
```

## 3. Transmitting images

Suppose we want to transmit the "cameraman" image over a communication channel. The intuitive way to do so is to send the  pixel values one by one, which corresponds to sending the coefficients of the decomposition of the image over the canonical basis. So far, nothing complicated: to send the cameraman image, for instance, we will send $64\times 64 = 4096$ integer values in a row.

Now suppose that a communication failure takes place after the first half of the pixels have been sent. The received data will allow us to display an approximation of the original image only. If we replace the missing data with zeros, here is what we would see, which is not very pretty:

```python
# unrolling of the image for transmission (we go column by column, hence "F")
tx_img = np.ravel(img, "F")

# oops, we lose half the data
tx_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.reshape(tx_img, (64, 64), "F")
plt.matshow(rx_img);
```

Can we come up with a trasmission scheme that is more robust in the face of channel loss? Interestingly, the answer is yes, and it involves a different, more versatile basis for the space of images.

### 3.1. The search for a good basis

What we are after is a basis for the space of images that fulfills a set of prerequisites:

 * we want the basis to be orthonormal (or at least orthogonal) so that basis decomposition is obtained simply via a series of inner products
 * we want the basis to be able to represent the image information robustly so that a few, important coefficients will capture most of the image; this will ensure resilence against data loss
 * we want the basis to be easy to compute.

One such basis is the **Haar basis**. While we cannot go into too many details in this notebook, the curious will find a good starting point [here](https://chengtsolin.wordpress.com/2015/04/15/real-time-2d-discrete-wavelet-transform-using-opengl-compute-shader/). Mathematical formulas aside, the Haar basis works by encoding the information in a *hierarchical* way: the first basis vectors encode the broad information and the higher coefficients encode the detail.

First of all, to keep things simple, we will remain in the space of square matrices whose size is a power of two. The code to generate the Haar basis matrices is the following: first we generate a 1D Haar vector and then we obtain the basis matrices by taking the outer product of all possible 1D vectors (don't worry if it's not clear, the results are what's important):

```python
def haar1D(n, SIZE):
    # check power of two
    if math.floor(math.log(SIZE) / math.log(2)) != math.log(SIZE) / math.log(2):
        print("Haar defined only for lengths that are a power of two")
        return None
    if n >= SIZE or n < 0:
        print("invalid Haar index")
        return None

    # zero basis vector
    if n == 0:
        return np.ones(SIZE)

    # express n > 1 as 2^p + q with p as large as possible;
    # then k = SIZE/2^p is the length of the support
    # and s = qk is the shift
    p = math.floor(math.log(n) / math.log(2))
    pp = int(pow(2, p))
    k = SIZE / pp
    s = (n - pp) * k

    h = np.zeros(SIZE)
    h[int(s):int(s+k/2)] = 1
    h[int(s+k/2):int(s+k)] = -1
    # these are not normalized
    return h

def haar2D(n, SIZE=8):
    # get horizontal and vertical indices
    hr = haar1D(n % SIZE, SIZE)
    hv = haar1D(int(n / SIZE), SIZE)
    # 2D Haar basis matrix is separable, so we can
    #  just take the column-row product
    H = np.outer(hr, hv)
    H = H / math.sqrt(np.sum(H * H))
    return H
```

Let's look at a few basis matrices; note that matrices can have both positive and negative values, so that positive values will be white, negative values will be black and the value of zero will be represented as gray:

```python
multishow(haar2D(1), haar2D(2), haar2D(3), haar2D(62), haar2D(63));
```

We can notice two key properties

* each basis matrix has positive and negative values in some symmetric pattern: this means that the basis matrix will implicitly compute the difference between image areas
* low-index basis matrices take differences between large areas, while high-index ones take differences in smaller **localized** areas of the image

We can immediately verify that the Haar matrices are orthogonal:

```python
# let's use an 8x8 space; there will be 64 basis vectors
# compute all possible inner product and only print the nonzero results
for m in range(0,64):
    for n in range(0,64):
        r = np.sum(haar2D(m, 8) * haar2D(n, 8))
        if r != 0:
            print("[%dx%d -> %f] " % (m, n, r), end="")
```

### 3.3. Haar image decomposition

Although we know it from the properties of orthogonal bases, let us verify that we can decompose the Cameraman image onto the Haar basis and reconstruct it:

```python
# project the image onto the Haar basis, obtaining a vector of 4096 coefficients
# this is simply the analysis formula for the vector space with an orthogonal basis
tx_img = np.zeros(64*64)
for k in range(0, (64*64)):
    tx_img[k] = np.sum(img * haar2D(k, 64))

# now rebuild the image with the synthesis formula; since the basis is orthonormal
#  we just need to scale the basis matrices by the projection coefficients
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += tx_img[k] * haar2D(k, 64)

multishow(tx_img.reshape(64,64), rx_img);
```

As you can see, the set of Haar coefficients do not make much sense visually, but the decomposition works!

### 3.3. Transmission error with the Haar decomposition

Now let's see what happens if, like before, we lose the second half of the coefficients:

```python
# oops, we lose half the data
lossy_img = np.copy(tx_img);
lossy_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

That's quite remarkable, no? We've lost the same amount of information as before but the image is still acceptable. This is because we lost the coefficients associated to the fine details of the image but we retained the "broad strokes" encoded by the first half.

Note that if we lose the first half of the coefficients the result is markedly different:

```python
lossy_img = np.copy(tx_img);
lossy_img[0:int(len(tx_img)/2)] = 0

rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

In fact, schemes like this one are used in *progressive encoding*: send the most important information first and add details if the channel permits it. You may have experienced this while browsing the interned over a slow connection.

All in all, a great application of a change of basis!

# ---------------------------------

### Did you like this Notebook?
Yes, no, maybe? Why don't you give us some feedback using the completely anonymous form below? Thank you!

```python
from IPython.display import IFrame
IFrame('https://www.surveymonkey.com/r/NOTOSURVEY?notebook_set=COM303&notebook_id=ImageHaarBasis', 600, 800)
```

```python

```


```python
%matplotlib inline
from matplotlib import pyplot as pp
import numpy as np
```

# Introduction

Let's assume that we are given the function $f(\mathbf{x}) : \mathbb{R}^M \rightarrow \mathbb{R}$. At each point $\mathbf{x}$ this function produces the value $y = f(\mathbf{x})$. Due to real world circumstances, this assignment is usually noisy, meaning that for a point $\mathbf{x}$ rather than measuring $y$, we obtain a slightly misplaced value $\hat{y}$ calculated as
\begin{equation}
    \hat{y} = f(\mathbf{x}) + \mathcal{N}(0,\sigma^2).
\end{equation}
Here $\mathcal{N}(0,\sigma^2)$, is the normal distribution with mean $0$ and standard deviation $\sigma^2$. Here we will refere to each $(\mathbf{x},\hat{y})$ as a data point. This term is referred as Gaussian noise. Now given a set of data points $\{(\mathbf{x}_n,\hat{y}_n)\}_{n=1}^{N}$, our goal is to find a function $g(\mathbf{x}) : \mathbb{R}^M \rightarrow \mathbb{R}$ that approximates $f$ as close as possible. In other words, we would like to find the function $g$ such that
\begin{equation}
    \| g(\mathbf{x}) - f(\mathbf{x}) \|^2
\end{equation}
is minimized.

```python
# This class contains the dataset

class dataset:
    def _build_dataset( self ):
        self._content = []
        for i in range( self.N ):
            x = np.random.uniform( self.I0, self.I1 )
            y = self.func( x )
            y_n = y + np.random.normal( 0, self.noise_std )
            self._content.append( [ x,y,y_n ] )
        self._content = np.array( self._content )

    def _build_dataset_dense( self ):
        x = np.arange( self.I0, self.I1, ( self.I1 - self.I0 )/(self.N_dense+1) )
        y = self.func( x )

        arr = np.array( [ x, y ] )
        self._content_dense = arr.transpose()

    def __init__( self ):
        self.func = np.sin
        self.N = 20
        self.N_dense = 100
        self.noise_std = 0.2

        self.I0 = -1*np.pi
        self.I1 = np.pi

        self._build_dataset()
        self._build_dataset_dense()

    @property
    def x( self ):
        return self._content[:,0].ravel()

    @property
    def y( self ):
        return self._content[:,1].ravel()

    @property
    def y_n( self ):
        return self._content[:,2].ravel()

    @property
    def x_dense( self ):
        return self._content_dense[:,0].ravel()

    @property
    def y_dense( self ):
        return self._content_dense[:,1].ravel()

dset = dataset()
```

# Sinus Curve

Let $f(x) = \sin(x)$ for the values in the interval $[-\pi,\pi]$. The goal of this section is to reproduce the red data points given the blue plot data points. To achieve this goal we will be looking at two different ways of modeling the data.

```python
pp.plot( dset.x_dense, dset.y_dense,'k-.', label='Actual Curve' )
pp.plot( dset.x, dset.y, 'r.', label='Actual Values')
pp.plot( dset.x, dset.y_n, 'b.', label='Noisy Values')
pp.legend()
pp.grid()
```

## Polynomial Curve-Fitting

We can assume that $g$ belongs to the class of polynomial functions of the degree $D$. This gives $g$ the form of
\begin{equation}
    g(x) = \sum_{d=0}^{D} a_d x^d.
\end{equation}
To find $g$ we have to determine that values of $a_0, \dots, a_D$. For each $x_n$ we have the following equation
\begin{equation}
    \sum_{d=0}^{D} a_d x_n^d = \hat{y}_n,
\end{equation}
and this yields to a system of linear equations from which values of $a_0, \dots, a_D$ can be calculated. We assume that this system has the form of $Xa=Y$ and the matrices $X \in \mathbb{R}^{(N,D)}$ and $Y\in \mathbb{R}^{(N,1)}$ as calculated as following :

```python
D = 5

X = np.zeros((dset.N,D+1))
Y = np.zeros((dset.N,1))

for i in range( dset.N ):
    Y[i,0] = dset.y_n[i]
    for d in range( D+1 ):
        X[i,d] = dset.x[i] ** d

X = np.matrix( X )
Y = np.matrix( Y )
```

One solution to this system is obtained by $a = (X^TX)^{-1}X^{T}Y$.

```python
a = np.linalg.inv( X.T * X ) * X.T * Y
a = np.array( a )
a = a.ravel()
print( a )
```

    [-0.00933626  1.0647227  -0.04343132 -0.17778385  0.00544987  0.00821083]

Now given $a$ for every $x$ we can calculate the value of $g(x)$. We will do so for the values of x_dense.

```python
def g(x,D,a) :
    o = 0.0
    for d in range(D+1):
        o += a[d] * x ** d
    return o

y_pred = []
for p in dset.x_dense :
    y_pred.append( g(p,D,a) )
y_pred = np.array( y_pred )

pp.plot( dset.x_dense,dset.y_dense,'k-.', label='Actual Curve' )
pp.plot( dset.x_dense, y_pred, 'r-.', label='Predicted Values')
pp.legend()
pp.grid()
```

### Questions

1. How does changing $N$ and $D$ change the shape of the predicted curve?
2. Can we do this for some other functions? Lienar/Non-linear
3. How else can we obtain $a$?
4. How can we measure the error of this prediction?

## Radial Basis Function Kernel Curve Fitting (RBF Kernel)

Similar to polynomial curve fitting, our goal here to obtain the parameters of the predicted curve by solving a system of linear equations. Here, we will solve this problem by placing a basis function at each data point and formulate the predicted curve as a weighted sum of the basis functions. The Radial Basis funciton kernel has the form of
\begin{equation}
    k(x,x') = \exp(- \frac{\|x-x'\|^2}{2\sigma^2}).
\end{equation}
At each point the shape of this kernel is as following :

```python
def rbf( x, x_base, sigma2 ):
    return np.exp(-1* (( x-x_base )**2) / (2*sigma2) )

kernal_sigma2 = 0.5
x_base = 0

y_rbf = []
for x in dset.x_dense :
    y_rbf.append( rbf(x, x_base, kernal_sigma2) )

pp.plot( dset.x_dense,y_rbf,'k-.', label='RBF Kernel' )
pp.legend()
pp.grid()
```

Placing this function at each data point and calculating a weighted sum will give us
\begin{equation}
    g(x) = \sum_{n=1}^{N} a_n k(x,x_n).
\end{equation}
This sum again provides us with a system of linear equations with the form a $Ka=Y$ where $K \in \mathbb{R}^{(N,N)}$ and $Y \in \mathbb{R}^{(N,1)}$ and they are calculated as :

```python
K = np.zeros((dset.N,dset.N))
Y = np.zeros((dset.N,1))

for i in range( dset.N ):
    Y[i,0] = dset.y_n[i]
    for j in range( dset.N ):
        K[i,j] = rbf( dset.x[i], dset.x[j], kernal_sigma2 )

# Regularizer
K = K + np.eye(dset.N)

K = np.matrix( K )
Y = np.matrix( Y )
```

Similarly, we solve this system as $a = (K^TK)^{-1}K^{T}Y$.

```python
a = np.linalg.inv( K.T * K ) * K.T * Y
a = np.array( a )
a = a.ravel()
print( a )
```

    [ 0.22360265 -0.4208425  -0.12855211 -0.0325646   0.07441013 -0.00932151
      0.08161617  0.23760018 -0.33090978 -0.02289759  0.23674976 -0.29165578
     -0.17067804 -0.01623991  0.41249722 -0.09758318 -0.23785424 -0.13231298
      0.28005175  0.21525552]

Now given $a$ for every $x$ we can calculate the value of $g(x)$. We will do so for the values of x_dense.

```python
def g(x,x_basis,a) :
    o = 0.0
    for d in range(dset.N):
        o += a[d] * rbf(x,x_basis[d],kernal_sigma2)
    return o

y_pred = []
for x in dset.x_dense :
    y_pred.append( g(x,dset.x,a) )
y_pred = np.array( y_pred )

pp.plot( dset.x_dense,dset.y_dense,'k-.', label='Actual Curve' )
pp.plot( dset.x_dense, y_pred, 'r-.', label='Predicted Values')
pp.legend()
pp.grid()
```

### Questions

1. How does changing $\sigma^2$ changes the shape of the predicted curve?
2. What other basis functions can we use?
3. How can we measure the error of this prediction?


# Divide and Conquer

## Master Theorem

$$ T(n) = aT(\frac{n}{b}) + O(n^k) $$

So...

$$
T(n) = \begin{equation}
\left\{
  \begin{aligned}
    O(n^k)           && a  b^k
  \end{aligned}
  \right.
\end{equation}
$$

## Max element in list

```python
import random

a = list(range(30))
random.shuffle(a)
print(a)
```

    [13, 17, 9, 10, 5, 22, 15, 24, 14, 21, 23, 1, 11, 0, 12, 27, 7, 8, 20, 6, 25, 19, 29, 2, 4, 16, 3, 18, 26, 28]

### Brute Force

```python
def bfmax(a):
  m = a[0]
  for i in range(1, len(a)):
    if a[i] > m:
      m = a[i]
  return m
```

```python
res = bfmax(a)
assert res == 29
print(res)
```

    29

$ O(n) $

### Divide and conquer

```python
def dcmax(a, i, f):
  if i == f:
    return a[i]
  mid = (i + f) // 2
  maxleft = dcmax(a, i, mid)
  maxright = dcmax(a, mid + 1, f)
  return maxleft if maxleft > maxright else maxright
```

en c/c++

```c++
expr ? a : b
```

en python

```python
a if expre else b
```

```python
res = dcmax(a, 0, len(a) - 1)
assert res == 29
print(res)
```

    29

$$ T(n) = 2T(\frac{n}{2}) + O(1) $$

so we have:

$$
a = 2\\
b = 2\\
k = 0
$$

Then using the master theorem:

$$
T(n) = n^{\log_2{2}}\\
T(n) = n\\
T(n) \implies O(n)
$$

```python
bigboy = list(range(10000))
random.shuffle(bigboy)
%timeit bfmax(bigboy)
%timeit dcmax(bigboy, 0, 9999)
```

    1000 loops, best of 5: 769 µs per loop
    100 loops, best of 5: 4.51 ms per loop

## Sumarize list elements

```python
def _sumarize(a, i, f):
  if i == f:
    return a[i]
  else:
    mid = (i + f) // 2
    s1 = _sumarize(a, i, mid)
    s2 = _sumarize(a, mid + 1, f)
    return s1 + s2

def sumarize(a):
  return _sumarize(a, 0, len(a) - 1)
```

```python
res = sumarize(a)
assert res == sum(a)
res
```

    435

## Matrix Multiplication

for matrixes where $n = 2^k \,,\, k \in N$

### Classic way

```python
def matmul(a, b):
  n = len(a)
  c = [[0]*n for _ in range(n)]
  for i in range(n):
    for j in range(n):
      accum = 0
      for k in range(n):
        accum += a[i][k] * b[k][j]
      c[i][j] = accum
  return c
```

```python
import numpy as np
```

```python
n = 8
a = np.array(list(range(n*n))).reshape((n, n))
b = np.array(list(range(n*n))).reshape((n, n))
print(a)
print(b)
```

    [[ 0  1  2  3  4  5  6  7]
     [ 8  9 10 11 12 13 14 15]
     [16 17 18 19 20 21 22 23]
     [24 25 26 27 28 29 30 31]
     [32 33 34 35 36 37 38 39]
     [40 41 42 43 44 45 46 47]
     [48 49 50 51 52 53 54 55]
     [56 57 58 59 60 61 62 63]]
    [[ 0  1  2  3  4  5  6  7]
     [ 8  9 10 11 12 13 14 15]
     [16 17 18 19 20 21 22 23]
     [24 25 26 27 28 29 30 31]
     [32 33 34 35 36 37 38 39]
     [40 41 42 43 44 45 46 47]
     [48 49 50 51 52 53 54 55]
     [56 57 58 59 60 61 62 63]]

```python
c = matmul(a, b)
np.array(c)
```

    array([[ 1120,  1148,  1176,  1204,  1232,  1260,  1288,  1316],
           [ 2912,  3004,  3096,  3188,  3280,  3372,  3464,  3556],
           [ 4704,  4860,  5016,  5172,  5328,  5484,  5640,  5796],
           [ 6496,  6716,  6936,  7156,  7376,  7596,  7816,  8036],
           [ 8288,  8572,  8856,  9140,  9424,  9708,  9992, 10276],
           [10080, 10428, 10776, 11124, 11472, 11820, 12168, 12516],
           [11872, 12284, 12696, 13108, 13520, 13932, 14344, 14756],
           [13664, 14140, 14616, 15092, 15568, 16044, 16520, 16996]])

### Divide and conquer way

```python
def dcmatmul(a, b, c, ri, rf, ci, cf):
  n = len(a)
  if ri == rf:
    accum = 0
    for k in range(n):
      accum += a[ri][k] * b[k][cf]

    c[ri][cf] = accum
  else:
    rmid = (ri + rf) // 2
    cmid = (ci + cf) // 2
    dcmatmul(a, b, c, ri, rmid, ci, cmid)
    dcmatmul(a, b, c, rmid+1, rf, ci, cmid)
    dcmatmul(a, b, c, ri, rmid, cmid +1, cf)
    dcmatmul(a, b, c, rmid+1, rf, cmid + 1, cf)
```

```python
n = 8
a = np.array(list(range(n*n))).reshape((n, n))
b = np.array(list(range(n*n))).reshape((n, n))
c = np.zeros((n, n))
dcmatmul(a, b, c, 0, n-1, 0, n-1)
print(c)
```

    [[ 1120.  1148.  1176.  1204.  1232.  1260.  1288.  1316.]
     [ 2912.  3004.  3096.  3188.  3280.  3372.  3464.  3556.]
     [ 4704.  4860.  5016.  5172.  5328.  5484.  5640.  5796.]
     [ 6496.  6716.  6936.  7156.  7376.  7596.  7816.  8036.]
     [ 8288.  8572.  8856.  9140.  9424.  9708.  9992. 10276.]
     [10080. 10428. 10776. 11124. 11472. 11820. 12168. 12516.]
     [11872. 12284. 12696. 13108. 13520. 13932. 14344. 14756.]
     [13664. 14140. 14616. 15092. 15568. 16044. 16520. 16996.]]

```python
import pdb

def dcmatmul(a, b, c, ri, rf, ci, cf):
  n = len(a)
  #pdb.set_trace()
  if ri == rf and ci == cf:
    accum = 0
    for k in range(n):
      accum += a[ri][k] * b[k][cf]

    c[ri][cf] = accum
  elif ri == rf:
    cmid = (ci + cf) // 2
    dcmatmul(a, b, c, ri, rf, ci, cmid)
    dcmatmul(a, b, c, ri, rf, cmid + 1, cf)
  elif ci == cf:
    rmid = (ri + rf) // 2
    dcmatmul(a, b, c, ri, rmid, ci, cf)
    dcmatmul(a, b, c, rmid+1, rf, ci, cf)
  else:
    rmid = (ri + rf) // 2
    cmid = (ci + cf) // 2
    dcmatmul(a, b, c, ri, rmid, ci, cmid)
    dcmatmul(a, b, c, rmid+1, rf, ci, cmid)
    dcmatmul(a, b, c, ri, rmid, cmid +1, cf)
    dcmatmul(a, b, c, rmid+1, rf, cmid + 1, cf)
```

```python
a = np.array(list(range(9))).reshape(3, 3)
b = np.array(list(range(9))).reshape(3, 3)
c = matmul(a, b)
np.array(c)
```

    array([[ 15,  18,  21],
           [ 42,  54,  66],
           [ 69,  90, 111]])

```python
c = np.zeros((3, 3))
dcmatmul(a, b, c, 0, 2, 0, 2)
c
```

    array([[ 15.,  18.,  21.],
           [ 42.,  54.,  66.],
           [  0.,  90., 111.]])

```python

```


# Catalan Numbers and Dyck Words

One of the, I suspect, most important ways in which Catalan Numbers emerge is that they give the count of Dyck Words.

## Dyck Words

Dyck Words are strings consisting of two symbols, for example '[' and ']', so that there is an equal number of '[' and ']', and at any given point there are at least as many '[' as there are ']'. In this case, Dyck words of length $2n$ are the different ways in which $n$ pairs of balanced brackets can be arranged.

I.e.:

'[[][]]' is a Dyck word.

'[]][][' is not a Dyck word.

The number of Dyck words of length $2n$ is given by the Catalan number $C_n = \frac{1}{n+1}\left(\begin{array}{c}2n\\n\end{array}\right)$.

## Catalan Numbers

Wikipedia has a few ways for deriving the Catalan numbers. I found it useful to just systematically count paths. You go through the grid and you label the edges by keeping track of "inflow". This doesn't easily get you to a closed-form solution, but it generalizes well for these lattice-path type of problems.

You can label each of the grid points with the amount of paths flowing through them:

\begin{equation}
f(x,y) = f(x-1,y) + f(x,y-1)\\ \forall x\geq y
\end{equation}

\begin{equation}
f(x,y) = 0\ \mathrm{otherwise}
\end{equation}

Which works out to:

\begin{equation}
f(x,y) = \sum^x_{k=y}f(k,y-1)
\end{equation}

With boundary condition:

\begin{equation}
f(x,0) = 1\\ \forall x\geq y
\end{equation}

\begin{equation}
f(x,0) = 0\ \mathrm{otherwise}
\end{equation}

```python
import numpy as np

def print_grid_long(N):
    """
    Print the path density on the grid. (long formula)
    """
    Grid = np.zeros([N,N])
    for x in range(Grid.shape[0]):
        for y in range(Grid.shape[1]):

            # boundary
            if y == 0:
                if x >= 0:
                    Grid[x,y] = 1

            elif y <= x:
                Grid[x,y] = int(Grid[x-1,y] + Grid[x,y-1])

            else:
                pass

    f = lambda s : str(s) if s!=0 else '-'
    y_len = len(Grid[0])
    for x in Grid:
        print(y_len*'%s\t' % tuple([f(int(i))  for i in x]))

    return Grid

def print_grid_short(N):
    """
    Print the path density on the grid. (short formula)
    """
    Grid = np.zeros([N,N])
    for x in range(Grid.shape[0]):
        for y in range(Grid.shape[1]):

            # boundary
            if y == 0:
                if x >= 0:
                    Grid[x,y] = 1

            else:
                Grid[x,y] = np.sum(Grid[y:x+1,y-1])

    f = lambda s : str(s) if s!=0 else '-'
    y_len = len(Grid[0])
    for x in Grid:
        print(y_len*'%s\t' % tuple([f(int(i))  for i in x]))

    return Grid

Grid = print_grid_short(16)
Grid[Grid == 0] = np.nan

plt.figure(figsize=(12,10))
plt.imshow(np.log10(Grid))
plt.colorbar()
plt.title('Path Flow (log10)')
```

```python
import matplotlib.pyplot as plt
import numpy as np

def dyck_words(n):
    """
    return all dyck words of length n
    """
    assert n%2 == 0 # has to be an even number

    if n == 0:
        return ''

    elif n == 2:
        yield 'XY'

    else:
        W = 'X'
        X_count = 1

        for letter in 'YX':

            if letter == 'Y':
                for word in dyck_words(n-2):
                    # equal to XY + Dyck(n-2)
                    yield W +'Y' + word

            elif letter == 'X' and X_count < n/2:
                X_count += 1
                results = set()
                for word in dyck_words(n-2):

                    """
                    add a Y somewhere after any X of the remaining dyck words

                    this works, but it creates duplicates that will fill up memory.

                    will have to figure out something smarter when there's time.

                    """

                    for i in range(n-2):
                        if word[i] == 'X':
                            res = W + word[:i+1]+'Y'+word[i+1:]
                            if res not in results:
                                results.update([res])
                                yield res
                            else:
                                pass

def factorial(x):
    if x == 0:
        return 1
    else:
        res = 1
        for i in range(1,x+1):
            res *= i
    return res

def catalan(n):
    return int((1/(n+1))*(factorial(2*n)/(factorial(n)**2)))

def plot_mountain_range(n):
    """
    One way to illustrate Dyck Words is to look at 'mountain ranges' of length n=2k,
    where mountains never dip below the horizon
    """

    code = {'X':1,'Y':-1}

    fig = plt.figure(figsize=(12,7))
    ax = plt.subplot()

    offset = 0

    words = sorted(list(dyck_words(n)))[::-1]

    for word in words:
        mountains = [code[letter] for letter in word]

        ax.plot(offset+np.hstack([[0],np.cumsum(mountains)]),color='royalblue',alpha=0.1)

        offset += 0.002

    #ax.axes.get_xaxis().set_visible(False)
    ax.axes.get_yaxis().set_visible(False)

    pass

```

```python
plot_mountain_range(14)
```

```python
for k in range(10):
    n = 2*k

    count = 0
    for word in dyck_words(n):
        count += 1

    cn = catalan(k)

    print('2n=%i Catalan_n: %i Dyck words: %i' % (n,cn,count))
```

    2n=0 Catalan_n: 1 Dyck words: 0
    2n=2 Catalan_n: 1 Dyck words: 1
    2n=4 Catalan_n: 2 Dyck words: 2
    2n=6 Catalan_n: 5 Dyck words: 5
    2n=8 Catalan_n: 14 Dyck words: 14
    2n=10 Catalan_n: 42 Dyck words: 42
    2n=12 Catalan_n: 132 Dyck words: 132
    2n=14 Catalan_n: 429 Dyck words: 429
    2n=16 Catalan_n: 1430 Dyck words: 1430
    2n=18 Catalan_n: 4862 Dyck words: 4862

```python
# just checking.

a = print_grid_long(15)
b = print_grid_short(15)
assert np.all(a == b)
```

    1	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    1	1	-	-	-	-	-	-	-	-	-	-	-	-	-
    1	2	2	-	-	-	-	-	-	-	-	-	-	-	-
    1	3	5	5	-	-	-	-	-	-	-	-	-	-	-
    1	4	9	14	14	-	-	-	-	-	-	-	-	-	-
    1	5	14	28	42	42	-	-	-	-	-	-	-	-	-
    1	6	20	48	90	132	132	-	-	-	-	-	-	-	-
    1	7	27	75	165	297	429	429	-	-	-	-	-	-	-
    1	8	35	110	275	572	1001	1430	1430	-	-	-	-	-	-
    1	9	44	154	429	1001	2002	3432	4862	4862	-	-	-	-	-
    1	10	54	208	637	1638	3640	7072	11934	16796	16796	-	-	-	-
    1	11	65	273	910	2548	6188	13260	25194	41990	58786	58786	-	-	-
    1	12	77	350	1260	3808	9996	23256	48450	90440	149226	208012	208012	-	-
    1	13	90	440	1700	5508	15504	38760	87210	177650	326876	534888	742900	742900	-
    1	14	104	544	2244	7752	23256	62016	149226	326876	653752	1188640	1931540	2674440	2674440
    1	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    1	1	-	-	-	-	-	-	-	-	-	-	-	-	-
    1	2	2	-	-	-	-	-	-	-	-	-	-	-	-
    1	3	5	5	-	-	-	-	-	-	-	-	-	-	-
    1	4	9	14	14	-	-	-	-	-	-	-	-	-	-
    1	5	14	28	42	42	-	-	-	-	-	-	-	-	-
    1	6	20	48	90	132	132	-	-	-	-	-	-	-	-
    1	7	27	75	165	297	429	429	-	-	-	-	-	-	-
    1	8	35	110	275	572	1001	1430	1430	-	-	-	-	-	-
    1	9	44	154	429	1001	2002	3432	4862	4862	-	-	-	-	-
    1	10	54	208	637	1638	3640	7072	11934	16796	16796	-	-	-	-
    1	11	65	273	910	2548	6188	13260	25194	41990	58786	58786	-	-	-
    1	12	77	350	1260	3808	9996	23256	48450	90440	149226	208012	208012	-	-
    1	13	90	440	1700	5508	15504	38760	87210	177650	326876	534888	742900	742900	-
    1	14	104	544	2244	7752	23256	62016	149226	326876	653752	1188640	1931540	2674440	2674440

Combinatorics - Catalan Numbers and Dyck Words.ipynb


# the German tank problem

the Germans have a population of $n$ tanks labeled with serial numbers $1,2,...,n$ on the tanks. The number of tanks $n$ is unknown and of interest to the Allied forces. The Allied forces randomly capture $k$ tanks from the Germans with replacement and observe their serial numbers $\{x_1, x_2, ..., x_k\}$. The goal is to, from observing the serial numbers on this random sample of the tanks, estimate $n$.

the *estimator* of $n$ maps an outcome of the experiment to an estimate of $n$, $\hat{n}$.

```julia
using StatsBase
using PyPlot
using Statistics
using Printf

PyPlot.matplotlib.style.use("seaborn-pastel")

rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16
```

    16

## data structure for a tank

for elegance

```julia
struct Tank
    serial_no::Int
end
```

## visualizing the captured tanks and their serial numbers

```julia
function viz_tanks(tanks::Array{Tank}, savename=nothing)
    nb_tanks = length(tanks)

    img = PyPlot.matplotlib.image.imread("tank.png")

    fig, ax = subplots(1, nb_tanks)
    for (t, tank) in enumerate(tanks)
        ax[t].imshow(img)
        ax[t].set_title(tank.serial_no)
        ax[t].axis("off")
    end
    tight_layout()

    if ! isnothing(savename)
        savefig(savename * ".png", format="png", dpi=300)
        # Linux command line tool to trim white space
        run(`convert $savename.png -trim $savename.png`)
    end
end

tanks = [Tank(rand(1:500)) for i = 1:5]
viz_tanks(tanks)
```

## simulating tank capture

write a function `capture_tanks` to simulate the random sampling of `nb_tanks_captured` tanks from all `nb_tanks` tanks the Germans have (without replacement). return a random sample of tanks.

```julia
function capture_tanks(nb_tanks_captured::Int, nb_tanks::Int)
    serial_nos = sample(1:nb_tanks, nb_tanks_captured, replace=false)
    return [Tank(serial_no) for serial_no in serial_nos]
end

captured_tanks = capture_tanks(5, 300)
viz_tanks(captured_tanks)
```

## defining different estimators

an estimator maps an outcome $\{x_1, x_2, ..., x_k\}$ to an estimate for $n$, $\hat{n}$.

### estimator (1): maximum serial number

this is the maximum likelihood estimator.

\begin{equation}
\hat{n} = \max_i x_i
\end{equation}

```julia
function max_serial_no(tanks::Array{Tank})
    serial_nos = [tank.serial_no for tank in tanks]
    return maximum(serial_nos)
end

max_serial_no(captured_tanks)
```

    269

### estimator (2): maximum serial number plus initial gap

\begin{equation}
\hat{n} = \max_i x_i + \bigl(\min_i x_i -1\bigr)
\end{equation}

```julia
function max_serial_no_plus_initial_gap(tanks::Array{Tank})
    serial_nos = [tank.serial_no for tank in tanks]
    return maximum(serial_nos) + minimum(serial_nos) - 1
end
```

    max_serial_no_plus_initial_gap (generic function with 1 method)

### estimator (3): maximum serial number plus gap if samples are evenly spaced

\begin{equation}
\hat{n} = \max_i x_i + \bigl( \max_i x_i / k -1 \bigr)
\end{equation}

```julia
function max_serial_no_plus_gap_if_evenly_spaced(tanks::Array{Tank})
    serial_nos = [tank.serial_no for tank in tanks]
    return maximum(serial_nos) * (1 + 1 / length(tanks)) - 1
end
```

    max_serial_no_plus_gap_if_evenly_spaced (generic function with 1 method)

## assessing the bias and variance of different estimators

say the Germans have `nb_tanks` tanks, and we randomly capture `nb_tanks_captured`. what is the distribution of the estimators (over different outcomes of this random experiment), and how does the distribution compare to the true `nb_tanks`?

```julia
"""
Perform `nb_sims` simulations of capturing `nb_tanks_captured` from `nb_tanks` total tanks w./o replacement.
Use the `estimator` function to map the captured array of tanks to an estimate of n in each simulation.
Return an array of estimated numbers of tanks from the simulations
"""
function sim_̂n(nb_sims::Int, nb_tanks::Int, nb_tanks_captured::Int, estimator::Function)
    # perform `nb_sims` of tank capture, collect estimates of n
    estimated_nb_tanks = zeros(nb_sims)
    for s = 1:nb_sims
        captured_tanks = capture_tanks(nb_tanks_captured, nb_tanks)
        estimated_nb_tanks[s] = estimator(captured_tanks)
    end
    return estimated_nb_tanks
end
```

    sim_̂n

```julia
nb_sims = 10000 # how many times we simulate tank capture to get the dist'n of the estimators

nb_tanks = 100 # n
nb_tanks_captured = 5 # k
```

    5

```julia
# an array of functions!
estimators = [max_serial_no, max_serial_no_plus_initial_gap, max_serial_no_plus_gap_if_evenly_spaced]

# loop over estimators
for e = 1:3
    estimated_nb_tanks = sim_̂n(nb_sims, nb_tanks, nb_tanks_captured, estimators[e])

    # plot histogram of estimator to viz its dist'n
    hist(estimated_nb_tanks, label="estimator ($e)",
        alpha=0.4, density=true, ec=@sprintf("C%d", e - 1))

    println("estimator ($e)")
    println("\tmean estimate: ", mean(estimated_nb_tanks))
    println("\tstandard deviation: ", std(estimated_nb_tanks))
end
xlabel("estimated # tanks")
ylabel("# simulations")
# plot true value for comparison
axvline(x=nb_tanks, color="k", linestyle="--", label="truth")
legend(prop=Dict("size" => 12))
```

estimator (1) is biased.

estimator (2) is unbiased but has a high variance.

estimator (3) is unbiased and has a low variance. it can be shown that estimator (3) is the minimum-variance, unbiased estimator!

## what happens as we capture more and more tanks, i.e. increase $k$?

assess estimator (3).

```julia
nb_sims = 10000 # how many times we simulate tank capture to get the dist'n of the estimators

nb_tanks = 100 # n
nb_tanks_captureds = [2, 5, 10, 20] # ks

color_norm = PyPlot.matplotlib.colors.Normalize(vmin=minimum(nb_tanks_captureds), vmax=maximum(nb_tanks_captureds))
k_to_color = PyPlot.cm.ScalarMappable(norm=color_norm, cmap=get_cmap("viridis")).to_rgba

for (i, nb_tanks_captured) in enumerate(nb_tanks_captureds)
    estimated_nb_tanks = sim_̂n(nb_sims, nb_tanks,
        nb_tanks_captured, max_serial_no_plus_gap_if_evenly_spaced)

    # plot histogram of estimator to viz its dist'n
    hist(estimated_nb_tanks, label="\$k=$nb_tanks_captured\$",
        alpha=0.5, density=true, color=k_to_color(nb_tanks_captured))
end
legend(loc="upper left")
xlabel("estimated # tanks")
ylabel("# simulations")
```

## one-sided confidence interval

how confident are we that the Germans don't have *more* tanks?

significance level: $\alpha$

test statistic = estimator (3) = $\hat{n} = \max_i x_i + \bigl( \max_i x_i / k -1 \bigr)$

**null hypothesis**: the number of tanks is $n=n_0$<br>
**alternative hypothesis**: the number of tanks is less than $n_0$

we reject the null hypothesis (say, "the data does not support the null hypothesis") that the number of tanks is $n=n_0$ if the p-value is less than $\alpha$. the p-value is the probability that, if the null hypothesis is true, we get a test statistic equal to or smaller than we observed.

we want to find the highest $n_0$ such that we have statistical power to reject the null hypothesis in favor of the alternative hypothesis. this is the upper bound on the confidence interval!

then the idea is that, if the null hypothesis is that the number of tanks is absurdly large compared to the largest serial number we saw in our sample, it would be very unlikely that we would see such small serial numbers compared to the number of tanks, so we'd reject the null hypothesis.

```julia
k = 5 # number of tanks to sample
n = 100 # pretend we don't know this

captured_tanks = capture_tanks(k, n)
viz_tanks(captured_tanks)

println("our estimate: ")
n̂ = max_serial_no_plus_gap_if_evenly_spaced(captured_tanks)
```

say $\alpha=0.05$ and we seek a 95% one-sided confidence interval

```julia
nb_sims = 100000
ns = 100:2:200
lower_cis = zeros(length(ns))

for (i, n) in enumerate(ns)
    estimated_nb_tanks = sim_̂n(nb_sims, n, k, max_serial_no_plus_gap_if_evenly_spaced)
    lower_cis[i] = percentile(estimated_nb_tanks, 5.0)
end

# find upper limit
idx_upper_limit_on_n̂ = findfirst(lower_cis .> n̂)
println("95% confidence upper limit on n̂ = ", ns[idx_upper_limit_on_n̂])

plot(ns, lower_cis, color="k", label="95% CI lower bound")
axhline(y=n̂)
title("lower bound on number of tanks")
xlabel("\$n\$")
ylabel("\$\\hat{n}\$, 95% CI")
legend()
```


# Conditional Probability - Dice Toss

> This document is written in *R*.
>
> ***GitHub***: https://github.com/czs108

## Background

> If you throw **2** dice, let the *total* score be $X$.

```R
scores <- matrix(nrow=6, ncol=6)
for (i in c(1:6)) {
    for (j in c(1:6)) {
        scores[i, j] <- i + j
    }
}

scores
```

2 	3 	4 	 5	 6	 7

3 	4 	5 	 6	 7	 8

4 	5 	6 	 7	 8	 9

5 	6 	7 	 8	 9	10

6 	7 	8 	 9	10	11

7 	8 	9 	10	11	12

```R
hist(scores, xlab="Score", ylab="Count", breaks=c(1.5:12.5))
```

## Question A

> What are the probabilities of each value of $X$?

```R
probs <- round(table(scores) / (6 * 6), 2)

probs
```

    scores
       2    3    4    5    6    7    8    9   10   11   12
    0.03 0.06 0.08 0.11 0.14 0.17 0.14 0.11 0.08 0.06 0.03

## Question B

\begin{equation}
\begin{split}
P(1\ of\ the\ scores = 4\, \mid\, X = 9)
    &= \frac{P(\{4, 5\}) + P(\{5, 4\})}{\sum_{sum(i, j) = 9} P(\{i, j\})} \\
    &= \frac{2}{4}
\end{split}
\end{equation}

## Question C

\begin{equation}
\begin{split}
P(X = 9\, \mid\, only\ 1\ score = 4)
    &= \frac{\sum_{sum(i, j) = 9} P(\{i, j\})}{\sum_{i = 1, i \neq 4}^{6} P(\{4, i\}) + \sum_{i = 1, i \neq 4} P(\{i, 4\})} \\
    &= \frac{4}{10}
\end{split}
\end{equation}

## Question D

\begin{equation}
\begin{split}
P(at\ least\ 1\ score = 4)
    &= \sum_{i = 1}^{6} P(\{4, i\}) + \sum_{i = 1}^{6} P(\{i, 4\}) - P({4, 4}) \\
    &= \frac{11}{36}
\end{split}
\end{equation}

## Question E

\begin{equation}
\begin{split}
P(2\ scores\ are\ the\ same)
    &= \sum_{i = 1}^{6} P(\{i, i\}) \\
    &= \frac{6}{36}
\end{split}
\end{equation}

## Question F

> What are the *mean* and *variance* of $X$?

```R
probs <- list("score"=as.numeric(names(probs)), "prob"=as.numeric(probs))

mean <- with(probs, sum(score * prob))
var <- with(probs, sum(((score - mean) ^ 2) * prob))

print(sprintf("Mean = %.2f", mean), quote=FALSE)
print(sprintf("Variance = %.2f", var), quote=FALSE)
```

    [1] Mean = 7.07
    [1] Variance = 6.02

exercises/Conditional Probability - Dice Toss.ipynb


```python
from sympy import sympify

expor = sympify("Matrix([[-b1*l1*cos(a1) - l2*(b1 + b2)*cos(a1 + a2) - l3*(b1 + b2 + b3)*cos(a1 + a2 + a3) - l4*(b1 + b2 + b3 + b4)*cos(a1 + a2 + a3 + a4), -l2*(b1 + b2)*cos(a1 + a2) - l3*(b1 + b2 + b3)*cos(a1 + a2 + a3) - l4*(b1 + b2 + b3 + b4)*cos(a1 + a2 + a3 + a4), -l3*(b1 + b2 + b3)*cos(a1 + a2 + a3) - l4*(b1 + b2 + b3 + b4)*cos(a1 + a2 + a3 + a4), -l4*(b1 + b2 + b3 + b4)*cos(a1 + a2 + a3 + a4)], [-b1*l1*sin(a1) - l2*(b1 + b2)*sin(a1 + a2) - l3*(b1 + b2 + b3)*sin(a1 + a2 + a3) - l4*(b1 + b2 + b3 + b4)*sin(a1 + a2 + a3 + a4), -l2*(b1 + b2)*sin(a1 + a2) - l3*(b1 + b2 + b3)*sin(a1 + a2 + a3) - l4*(b1 + b2 + b3 + b4)*sin(a1 + a2 + a3 + a4), -l3*(b1 + b2 + b3)*sin(a1 + a2 + a3) - l4*(b1 + b2 + b3 + b4)*sin(a1 + a2 + a3 + a4), -l4*(b1 + b2 + b3 + b4)*sin(a1 + a2 + a3 + a4)]])")
```

```python
expor
```

$\displaystyle \left[\begin{matrix}- b_{1} l_{1} \cos{\left(a_{1} \right)} - l_{2} \left(b_{1} + b_{2}\right) \cos{\left(a_{1} + a_{2} \right)} - l_{3} \left(b_{1} + b_{2} + b_{3}\right) \cos{\left(a_{1} + a_{2} + a_{3} \right)} - l_{4} \left(b_{1} + b_{2} + b_{3} + b_{4}\right) \cos{\left(a_{1} + a_{2} + a_{3} + a_{4} \right)} & - l_{2} \left(b_{1} + b_{2}\right) \cos{\left(a_{1} + a_{2} \right)} - l_{3} \left(b_{1} + b_{2} + b_{3}\right) \cos{\left(a_{1} + a_{2} + a_{3} \right)} - l_{4} \left(b_{1} + b_{2} + b_{3} + b_{4}\right) \cos{\left(a_{1} + a_{2} + a_{3} + a_{4} \right)} & - l_{3} \left(b_{1} + b_{2} + b_{3}\right) \cos{\left(a_{1} + a_{2} + a_{3} \right)} - l_{4} \left(b_{1} + b_{2} + b_{3} + b_{4}\right) \cos{\left(a_{1} + a_{2} + a_{3} + a_{4} \right)} & - l_{4} \left(b_{1} + b_{2} + b_{3} + b_{4}\right) \cos{\left(a_{1} + a_{2} + a_{3} + a_{4} \right)}\\- b_{1} l_{1} \sin{\left(a_{1} \right)} - l_{2} \left(b_{1} + b_{2}\right) \sin{\left(a_{1} + a_{2} \right)} - l_{3} \left(b_{1} + b_{2} + b_{3}\right) \sin{\left(a_{1} + a_{2} + a_{3} \right)} - l_{4} \left(b_{1} + b_{2} + b_{3} + b_{4}\right) \sin{\left(a_{1} + a_{2} + a_{3} + a_{4} \right)} & - l_{2} \left(b_{1} + b_{2}\right) \sin{\left(a_{1} + a_{2} \right)} - l_{3} \left(b_{1} + b_{2} + b_{3}\right) \sin{\left(a_{1} + a_{2} + a_{3} \right)} - l_{4} \left(b_{1} + b_{2} + b_{3} + b_{4}\right) \sin{\left(a_{1} + a_{2} + a_{3} + a_{4} \right)} & - l_{3} \left(b_{1} + b_{2} + b_{3}\right) \sin{\left(a_{1} + a_{2} + a_{3} \right)} - l_{4} \left(b_{1} + b_{2} + b_{3} + b_{4}\right) \sin{\left(a_{1} + a_{2} + a_{3} + a_{4} \right)} & - l_{4} \left(b_{1} + b_{2} + b_{3} + b_{4}\right) \sin{\left(a_{1} + a_{2} + a_{3} + a_{4} \right)}\end{matrix}\right]$