```python
%matplotlib inline
import matplotlib
import seaborn as sns
sns.set()
matplotlib.rcParams['figure.dpi'] = 144
```

# Probability

<!-- requirement: images/rv_flowchart.png -->

```python
import numpy as np
import scipy as sp
import pandas as pd
import matplotlib.pylab as plt
```

```python
from scipy.stats import norm

def plot_hist_dist(rvs, dist, title=None, label='', ax=None, line_color='black', bar_color='blue'):
    ax = ax if ax else plt.gca()
    _, bins, _ = ax.hist(rvs, bins=50, alpha=.6, density=True, label=(label + ' rvs').strip(), color=bar_color)
    xmin, xmax = bins.min(), bins.max()
    xpoints = np.arange(xmin, xmax, (xmax - xmin) / 100)
    ax.plot(xpoints, dist.pdf(xpoints), label=(label+' pdf').strip(), color=line_color)
    if title:
        ax.set_title(title)
    ax.legend()

def plot_hist_dist_discrete(rvs, dist, title=None, label='', ax=None, line_color='black', bar_color='blue'):
    ax = ax if ax else plt.gca()
    uniques = np.unique(rvs)
    mids = (uniques[1:] + uniques[:-1]) / 2.
    bins = np.hstack([[uniques[0]-.5], mids, [uniques[-1] + .5]])
    plt.hist(rvs, bins=bins, density=True, label=(label + ' rvs').strip(), alpha=.6, color=bar_color)
    if title:
        ax.set_title(title)
    plt.plot(uniques, dist.pmf(uniques), label=(label + ' pmf').strip(), color=line_color)
    ax.legend()

def print_mean_variance(rvs, dist):
    print("Mean:     Theoretical {:.4f}, Actual {:.4f}".format(dist.mean(), rvs.mean()))
    print("Variance: Theoretical {:.4f}, Actual {:.4f}".format(dist.var(),  rvs.var()))
```

```python
from ipywidgets import interact, FloatSlider, IntSlider
from functools import wraps

def gen_interact_dist(plotter):
    @wraps(plotter)
    def wrapper(dist, xlim, ylim, N=2000):
        def wrapped(**kwargs):
            _dist = dist(**kwargs)
            rvs = _dist.rvs(N)

            plotter(_dist, rvs, xlim)

            plt.title(_dist.dist.name)
            plt.xlim(xlim)
            plt.ylim(ylim)
            plt.legend()
        return wrapped
    return wrapper

@gen_interact_dist
def interact_dist(dist, rvs, xlim):
    """
    Function plots rvs and pdf given a dist, xlim, and ylim
    """
    x = np.linspace(*xlim, num=5000)
    p = dist.pdf(x)
    plt.hist(rvs, bins=50, density=True, alpha=0.6, color='blue', label='rvs')
    plt.plot(x, p, color='black', label='pdf')

@gen_interact_dist
def interact_dist_discrete(dist, rvs, xlim):
    """
    Function plots rvs and pmf given a dist, xlim, and ylim
    """
    x = np.linspace(*xlim, num=5000, dtype=int)
    p = dist.pmf(x)

    uniques = np.unique(rvs)
    mids = (uniques[1:] + uniques[:-1]) / 2.
    bins = np.hstack([[uniques[0]-.5], mids, [uniques[-1] + .5]])

    plt.hist(rvs, bins=bins, density=True, alpha=0.6, color='blue', label='rvs')
    plt.plot(x, p, color='black', label='pdf')

```

## Random Variables

Probability and statistics are based on the notion of a **random variable**.  Random variables are *abstract mathematical quantities* (often denoted $X$) which can take on values at random.  Random variates (`rvs`) are instances of random variables.

**Example:** Suppose we roll a standard fair die.  The number shown on top is a random variable taking on the values 1 through 6, inclusive.

While the value of a variate can be anything, we know what values tend to be based on a **distribution**, $p(x)$, which is the probability of $X$ taking on any value $x$.  Their distributions are how we reason about randomness mathematically.

**Example:** In the previous dice example, the dice rolls are distributed evenly ($\frac{1}{6}$ probability of each event) over all six possible values.

While people (including us) often use a random variable and its distribution interchangeably, these are *not* the same concept and the distinction is not semantic.

**Example:** Suppose we have two dice, one red and one blue but otherwise identical.  Their rolls would be represented by two different random variables.  These two random variables have the same distribution.

```python
plot_hist_dist(norm.rvs(size=2000), norm, title=norm.name)
```

## Statistics of Random Variables

### Discrete Random Variables

For discrete-valued random variables, we can plot the distribution (or more precisely, the probability mass function or **PMF**, $p(x)$, which denotes the probability that $X = x$) of a random variable.  The **PMF** has the critical properties that it is non-negative $(p(x) \ge 0)$ and
$$ \sum_x p(x) = 1 \,.$$

We can compute statistics about random variables using the expectation operator $\mathbb{E}$.  The expectation operator has the following definition:
$$ \mathbb{E}[f(x)] = \sum f(x) p(x) $$

A simple (and important) example is the mean
$$ \mathbb{E}[X] = \sum x p(x) $$

We can also calculate the variance
$$ \mbox{Var}[X] = \mathbb{E}\left[(X-\mathbb{E}[X])^2\right] = \sum x^2 p(x) - \left( \sum x p(x) \right)^2 \,. $$
The standard deviation is simply the square root of the variance.

We may also wish to calculate the probability $\mathbb{P}$ of different **events** associated with these random variables.  For example the cumulative distribution function or **CDF** of a random variable is associated with the events

$$ F_X(a) = \mathbb{P}[X \le a] = \sum_{x \le a} p(x)\,. $$

**Examples:** There are a number of important random variables.  The discrete ones have distributions such as:

1. Bernoulli Distribution
1. Binomial Distribution
1. Geometric Distribution
1. Poisson Distribution

### Continuous Random Variables

So far, we have only given the formulas for discrete-valued random variables.  Continuous random variables have a probability distribution function or **PDF**, also denoted $p(x)$.  A PDF has to be non-negative ($p(x) \ge 0$) and satisfy
$$ \int_{(-\infty, \infty)} p(x) dx = 1\,. $$

Similarly you will also hear about the cumulative distribution function or **CDF** defined as
$$ F_X(x) = \int_{(-\infty, x]} p(y)dy\,. $$

The probability density of $X = x$ is given by $p(x)$.  Put another way, the probability that $a < X \le b$ is given by the integral
$$ \mathbb{P}[a < X \le b] = \int_{(a, b]}p(x)dx = F_X(b) - F_X(a) \,. $$

We can extend the statistics of discrete random variables to continuous random variables by exchanging discrete sums for continuous integrals.

$$ \mathbb{E}[f(x)] = \int_{(-\infty, \infty)} f(x) p(x) dx $$

**Examples:** Continuous random variables may have distributions such as:

1. Beta Distribution
1. Exponential Distribution
1. Normal Distribution

**Exercise:** What would the corresponding formulas for mean and standard deviation look like for continuous-valued random variables?

```python
# Some examples of distributions

dists = (sp.stats.expon(),
         sp.stats.norm(),
         sp.stats.beta(a=5, b=10))
for k, dist in enumerate(dists):
    ax = plt.subplot(3,2,2*k+1)
    plot_hist_dist(dist.rvs(size=2000), dist, title=dist.dist.name, ax=ax)

    xs = np.linspace(*plt.xlim())
    ax = plt.subplot(3,2,2*k+2)
    ax.plot(xs, dist.cdf(xs), label='cdf')
    mean, var = dist.stats('mv')
    ax.plot([mean]*2, [0., 1.], label='mean')
    plt.text(mean, 0.5, 'std', ha='center', va='bottom')
    plt.annotate("", xy=(mean-np.sqrt(var), 0.5), xycoords='data',
                 xytext=(mean+np.sqrt(var), 0.5), textcoords='data',
                 arrowprops=dict(arrowstyle="|-|", lw=2, color='r'))

    ax.legend(loc='lower right')
    ax.set_title(dist.dist.name)
plt.tight_layout()
```

## Quantile Functions

The **quantile function** (denoted $q_X(u)$) is the inverse of the cumulative distribution function or CDF $F_X(x)$.  That is,

$$ q_X (F_X(x)) = x \qquad F_X(q_X(u)) = u$$

Notice that since the range of $F_X$ is only $(0,1)$, then $q_X$ is a function on $(0,1)$.

**Theorem:** If $U$ is a uniform random variable then $q_X(U)$ is has the same distribution as $X$.

```python
# Quantile functions to generate random variables

N = 1000

dists = (
    sp.stats.expon(scale=1/2.),
    sp.stats.beta(a=2., b=4.),
    sp.stats.norm(loc=2., scale=4.),
    sp.stats.chi2(df=4.)
)

rvs = sp.stats.uniform().rvs(size=N)
for k, dist in enumerate(dists):
    ax = plt.subplot(2,2,k+1)
    plot_hist_dist(dist.ppf(rvs), dist, title=dist.dist.name, ax=ax)
plt.suptitle('rvs from quantile functions')
plt.tight_layout()
```

## Modeling observations with random variables

Many phenomena in the world are modeled as random variables. By considering the characteristics of the phenomena we would like to model, we can assume an appropriate random variable. Such assumptions lay the groundwork for statistic inference about the world. We can use measurements of the observed phenomena to compute statistics and estimate parameters of our model.

The following flow-chart is a short guide to some of the more common random variables we might use as models of real-world phenomena.

Simply knowing whether your observed values will be discrete or continuous, bounded or unbounded, already puts constraints on what kinds of random variables we might use to model a given phenomenon. However, a random variable will often only be appropriate as a model if the phenomenon meets certain assumptions. We'll explore each of the above random variables in more depth and expand upon these assumptions.

## Bernoulli Trials

Bernoulli, binomial, and geometric random variables are all defined in terms of **Bernoulli trials**. A Bernoulli trial is an event that is either a success or a failure. For example, a coin flip results in heads or tails, and we might define heads as success and failure as tails.

Bernoulli trials are assumed to be _independent_. If we are considering a random variable based on several Bernoulli trials (i.e. binomial or geometric random variables), we might also require that the outcomes of the Bernoulli trials are _identically distributed_.

### Bernoulli Random Variable

The Bernoulli random variable is the simplest random variable in probability.  The value of the trial $X$ can take on either $0$ (failure) or $1$ (success) and it is parameterized by the value $\Pi$, which is the probability that it is a success.  The PMF is given by

$$ p(x) = \left\{ \begin{align} \Pi && \mbox{if }x=1 \\ 1-\Pi && \mbox{if }x = 0 \end{align} \right. \,.$$

**Examples:**
- We can think of $p$ as the probability a (biased) coin lands on heads after a flip.
- Whether a user clicks on a displayed advertisement.
- Whether a borrower defaults on a loan.

**Stats:**
- The mean is $\mathbb{E}[X] = \Pi$
- The variance is $\mbox{Var}[X] = \Pi(1-\Pi)$.

```python
# Mean and variance of the Bernoulli

p = .4

dist = sp.stats.bernoulli(p)
rvs = dist.rvs(size=1000)
print_mean_variance(rvs, dist)
```

### Independence and Conditional Expectation

Two random variables are independent if knowing the results of one does not tell you anything about the other.  That is, if $p_{X, Y}(x,y)$ is the joint PMF or PDF for $(X,Y)$ and $p_X(x)$ and $p_Y(y)$ are the PMF or PDF for $X$ and $Y$, then a variable is independent if and only if

$$ p_{X, Y}(x,y) = p_X(x)p_Y(y)\,. $$

The **covariance** of two random variables $X$ and $Y$ is given by

$$ \begin{split} \mbox{Cov}[X,Y] &= \mathbb{E}\left[(X - \mathbb{E}[X]) (Y - \mathbb{E}[Y])\right] \\
&= \mathbb{E}[XY] - \mathbb{E}\left[X \,\mathbb{E}[Y]\right] - \mathbb{E}\left[Y \,\mathbb{E}[X]\right] + \mathbb{E}[X]\mathbb{E}[Y] \\
&= \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y] \end{split} $$

If two variables are independent, their *covariance* is zero, or equivalently the expectation of the product is just the product of their expectations
$$ \mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \,.$$

**Warning:** Just because two variables have zero covariance does not mean they are independent!  For example, if $X$ is a random variable taking on the values $\{-1, 0, 1\}$ with equal probability and $Y = X^2$, then they have zero covariance but are not independent.

When speaking of the joint distribution of random variables, it is often useful to speak of the **conditional expectation** given by
$$ \mathbb{E}[X \mid Y=y] = \sum_{x} x \frac{p_{X, Y}(x,y)}{p_Y(y)}\,. $$
It's useful to think of the above conditional expectation as a function of $y$.  We may also want to take the **conditional probability** of events related to $X$,
$$ \mathbb{P}[X \le a \mid Y=y] = \sum_{x \le a} \frac{p_{X, Y}(x,y)}{p_Y(y)}\,. $$

**Theorem:** If you have two independent random variables, $X$ and $Y$, the means add
$$ \mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] $$
and variances add
$$ \mbox{Var}[X + Y] = \mbox{Var}[X] + \mbox{Var}[Y]\,. $$

**Exercise:** What is the standard deviation of $X + Y$?
1. What is the standard deviation if you add $n$ independent and identically distributed random variables:
$$X_1 + \cdots + X_n$$
2. What is the standard deviation of the average:
$$ \frac{X_1 + \cdots + X_n}{n} $$

```python
# Independent and dependent Bernoulli random variables

p = .6
q = .9
r = p / (1-p) * (1-q)
N = 2500

# independent random variables
rvs_x1 = sp.stats.bernoulli(p).rvs(size=N)
rvs_y1 = sp.stats.bernoulli(p).rvs(size=N)

# dependent random variables
rvs_x2 = sp.stats.bernoulli(p).rvs(size=N)
rvs_y2 = sp.stats.bernoulli(np.where(rvs_x2, q, r)).rvs(size=N)

cov1 = np.cov(rvs_x1, rvs_y1)
cov2 = np.cov(rvs_x2, rvs_y2)

vmin = min(cov1.min(), cov2.min())
vmax = min(cov1.max(), cov2.max())

plt.subplot(2, 2, 1)
plt.matshow(cov1, fignum=False, vmin=vmin, vmax=vmax, cmap=plt.cm.Blues)
plt.grid(False)
plt.xlabel('X')
plt.ylabel('Y')
plt.title('independent cov', y=1.15)
plt.colorbar()

plt.subplot(2, 2, 2)
plt.matshow(cov2, fignum=False, vmin=vmin, vmax=vmax, cmap=plt.cm.Blues)
plt.grid(False)
plt.xlabel('X')
plt.ylabel('Y')
plt.title('dependent cov', y=1.15)
plt.colorbar()

plt.subplot(2, 2, 3)
pd.Series({
    'X=0': rvs_y1[rvs_x1==0].mean(),
    'X=1': rvs_y1[rvs_x1==1].mean(),
    'X=*': rvs_y1.mean(),
}).plot(kind='bar', title='conditional mean (independent)')
plt.ylim([0., 1.])

plt.subplot(2, 2, 4)
pd.Series({
    'X=0': rvs_y2[rvs_x2==0].mean(),
    'X=1': rvs_y2[rvs_x2==1].mean(),
    'X=*': rvs_y2.mean(),
}).plot(kind='bar', title='conditional mean (dependent)')
plt.ylim([0., 1.])

plt.tight_layout();
```

### Binomial Random Variable

The Binomial random variable is just the sum of $n$ independent Bernoulli random trials each with an independent probability $\Pi$ of success.  Its PMF is given by the binomial theorem:

$$ p(k) = \binom{n}{k} \Pi^k(1-\Pi)^{n-k}\,. $$

You can easily derive this formula by considering a coin with a probability, $\Pi$, of flipping heads and calculating the probability of seeing $k$ heads out of $n$ flips.

**Examples:**
- The total number of times a biased coin lands on heads after being flipped $n$ times.
- The total number of ad clicks after seeing $n$ ads if a user has an independent probability $\Pi$ of clicking on an advertisement.
- The number of points scored after $n$ free-throw attempts if each attempt has $\Pi$ probability of succeeding.

**Stats:**
- The mean is $\mathbb{E}[X] = n\Pi$
- The variance is $\mbox{Var}[X] = n\Pi(1-\Pi)$.

Notice that the mean increases with both $n$ and $\Pi$ but that the variance decreases with $\Pi$ for a fixed mean: $\mbox{Var}[X] = \mathbb{E}[X] \cdot (1-\Pi)$.  This is illustrated in the plots below.

```python
# Binomial distribution, two parameters

n_slider = IntSlider(value=10, min=5, max=50, step=1, description='n')
pi_slider = FloatSlider(value=0.5, min=0.2, max=0.8, step=0.05, description='$\Pi$')
interact(
    interact_dist_discrete(sp.stats.binom, xlim=[0, 18], ylim=[0, 0.3]),
    n=n_slider, p=pi_slider
);
```

**Theorem:** The sum of $n$ independent Bernoulli trials with probability $\Pi$ of success is a Binomial with parameters $n$ and $\Pi$.

**Exercise:** Prove the formulas for mean and variance using the fact that the Binomial is the sum.

```python
# Binomial is the sum of Bernoulli

p = .4
n = 6

rvs = sp.stats.bernoulli(p).rvs(size=[n, 1000]).sum(axis=0)
dist=sp.stats.binom(n, p)

plot_hist_dist_discrete(rvs, dist, title="Binomial")
```

### Geometric Distribution

The geometric distribution models the number $X$ of independent Bernoulli trials needed to obtain a success -- that is, we keep testing independent Bernoulli random variables until we are successful and measure the (random) number of such trials.  The PMF is given by

$$ p(k) = (1-\Pi)^{k-1}\Pi \,. $$

**Examples:**
- Suppose a chef needs to crack an egg without breaking a yolk.  If she keeps the yolks intact with probability $p$,  then $p(k)$ is the PMF over the number of eggs it takes.
- A newly-wed couple plans to have children until they have a girl.  The total number of children they expect to have is geometrically distributed.
- A person being treated for cancer is given chemotherapy drugs which have independent and identically distributed probability $\Pi$ of successful treatment.  The number of drugs he must take until he is successfully treated is geometrically distributed.

**Stats:**
- The mean is $\mathbb{E}[X] = \frac{1}{\Pi}$
- The variance is $\mbox{Var}[X] = \frac{1-\Pi}{\Pi^2}$.

```python
# The geometric distribution is the number of Bernoulli trials
# before a successful trial

p = .8

def geom(p):
    dist = sp.stats.bernoulli(p)
    k = 1
    while not dist.rvs():
        k += 1
    return k

rvs = np.zeros(shape=100)
for k in range(len(rvs)):
    rvs[k] = geom(p)

dist = sp.stats.geom(p)
plot_hist_dist_discrete(rvs, dist, title="Geometric Distribution")
```

As $\Pi$, the probability of success for any given trial, increases, we would expect the first success to occur early and the mean of the geometric random variable to be lower.  This is born out in the plots below.

```python
# Geometric distribution, one parameter

Pi_slider = FloatSlider(value=0.5, min=0.05, max=1.0, step=0.05, description='$\Pi$')
interact(
    interact_dist_discrete(sp.stats.geom, xlim=[1, 20], ylim=[0, 0.8]),
    p=Pi_slider
);
```

### Memoryless Property

The geometric distribution is _memoryless_.  Suppose your first successful Bernoulli trial is distributed $p(k)$.  Because the draws of the Bernoulli trials are independent, if you have $n$ failed trials, the distribution of *remaining* number of trials until success is still $p(k)$.  The process forgets its history. Past failures do not influence future probability of success.

**Examples:** After breaking $n$ eggs, the distribution of the number of eggs our chef must still crack is the same as when she started.

Mathematically, we observe that if $X$ is distributed as $p(k)$,
$$ P[X=(n+k) \mid X \ge n] = (1-\Pi)^{k-1}\Pi = P[X=k] $$

**Exercise:** Use the definition of conditional probability to prove the above statement.

```python
# The memoryless property of the geometric distribution

p = .4
n = 5
N = 2000

dist = sp.stats.geom(p)
rvs = dist.rvs(size=N)
plot_hist_dist_discrete(rvs[rvs > n] - n, dist, title="Conditional Geometric")
```

## Poisson Distribution

The Poisson distribution is a discrete-valued distribution representing the counts of independent events occurring within a fixed interval of time or space (which could be abstract, i.e not necessarily physical space).  The Poisson distribution describes an _arrival process_ (sometimes called a Poisson process), in which events occur at random with some fixed average rate. The Poisson distribution makes a few key assumptions:
- Independence: the occurrence of any single event does not affect the likelihood of another
- The events have the same likelihood of occurring on average
- The probability of an event occurring is proportional to the length of the interval

The distribution is parameterized by the rate parameter, $\lambda$, and has a PMF

$$ p(k) = \frac{\lambda^k e^{-\lambda}}{k!} $$

**Stats:**
- The mean is $\mathbb{E}[X] = \lambda$
- The variance is $\mbox{Var}[X] = \lambda$.

**Examples:**
- The number of goals scored in a World Cup match.
- The number of patient arrivals to an emergency room in one day.
- The number of asteroids hitting the earth each year.
- The number of Prussian Cavalry Officers kicked by their horse each year.

```python
# Poisson distribution, one parameter

lambda_slider = FloatSlider(value=2.0, min=2.0, max=8.0, step=0.05, description='$\lambda$')
interact(
    interact_dist_discrete(sp.stats.poisson, xlim=[0, 20], ylim=[0, 0.28]),
    mu=lambda_slider
);
```

## Exponential Distribution

The exponential distribution is a non-negative continuous random variable whose PDF is shaped exponentially

$$ p(x) = \lambda e^{-\lambda x} $$

The length of time (or space) between events in an arrival process are exponentially distributed. Therefore, the exponential distribution is closely related to the Poisson distribution, and they share the parameter, $\lambda$.

For example, imagine we track the arrival of customers at a store during a business day, and assume the customers arrive randomly with some average rate. When the store opens, we start a stopwatch, and when the first customer arrives we record the time on the stopwatch. We then reset the stopwatch and measure the length of time between the arrival of the first customer and the arrival of the second customer.

In this scenario, the times will be exponentially distributed with rate parameter, $\lambda$. The number of customers that arrive during opening hours will be Poisson distributed also with rate parameter, $\lambda$.

**Stats:**
- The mean is $\mathbb{E}[X] = \frac{1}{\lambda}$
- The variance is $\mbox{Var}[X] = \frac{1}{\lambda^2}$.

```python
# Exponential distribution, one parameter
# Note: scipy defines the distribution as exp(-(x-loc)/scale)/scale

scale_slider = FloatSlider(value=0.5, min=0.5, max=2.0, step=0.05, description='$1/\lambda$')
interact(
    interact_dist(sp.stats.expon, xlim=[0, 6], ylim=[0, 2]),
    scale=scale_slider
);
```

```python
# Poisson distribution as waiting times
lam = 3.5
T = 3.
N = 1000
M = 100

exp_rvs = sp.stats.expon(scale=1./lam).rvs(size=[N, M])
waiting_times_rvs = np.cumsum(exp_rvs, axis=1)
poisson_rvs = (waiting_times_rvs <= T).sum(axis=1)
plt.figure()
plot_hist_dist_discrete(poisson_rvs, sp.stats.poisson(mu=lam * T), title="Poisson 1")

poisson_rvs = np.bincount(np.floor(np.cumsum(exp_rvs) / T).astype(int))
plt.figure()
plot_hist_dist_discrete(poisson_rvs, sp.stats.poisson(mu=lam * T), title="Poisson 2")
```

You may notice that the PDF of the exponential distribution is similar to the PMF of the geometric distribution. The exponential distribution also has a memoryless property:

$$ \mathbb{P}[X=x+k \mid X \ge k]= \lambda e^{-\lambda x} = \mathbb{P}[X=x]$$

```python
# The memoryless property of the exponential distribution

lam=.2
a = 3.2
N = 2000

dist = sp.stats.expon(scale=1./lam)
rvs = dist.rvs(size=N)
plot_hist_dist(rvs[rvs >= a] - a, dist, title="Conditional Exponential")
```

## Normal Distribution

Probably the most famous distribution in all of statistics is the normal (or Gaussian) distribution.  It is parameterized by its mean $\mu$ and standard deviation $\sigma$, and the distribution is denoted $N(\mu, \sigma^2)$.  Its PDF is given by

$$ p(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left(\frac{(x - \mu)^2}{2 \sigma^2} \right) \,.$$

**Stats:**
- The mean is $\mathbb{E}[X] = \mu$
- The variance is $\mbox{Var}[X] = \sigma^2$.

**Examples:**
- The height or weight of students in a classroom.
- The distribution of test scores on the SAT.

**Useful Facts:** Let's write $n(x  \mid  \mu, \sigma^2)$ to be the normal PDF with mean $\mu$ and standard deviation $\sigma$.  A little bit of algebra shows that the distribution satisfies translational symmetry,

$$ n(x  \mid  \mu, \sigma^2) = n(x - \mu  \mid  0, \sigma^2) $$

and scaling symmetry

$$ n(x  \mid  \mu, \sigma^2) = n\left(\frac{x - \mu}{\sigma} \,\Big|\, 0, 1\right)\,. $$

A standard normal distribution is one with mean $\mu = 0$ and standard deviation $\sigma = 1$.

```python
# Normal distribution, two parameters

mu_slider = FloatSlider(value=0, min=-3, max=3, step=0.05, description='$\mu$')
sigma_slider = FloatSlider(value=1.0, min=0.5, max=2.0, step=0.05, description='$\sigma$')
interact(
    interact_dist(sp.stats.norm, xlim=[-6, 6], ylim=[0, 0.8]),
    loc=mu_slider, scale=sigma_slider
);
```

## Central Limit Distribution

Often, when we look at random variables, we want to understand their mean.  It turns out this behaves in a vary predictable way.  As $n \to \infty$,

$$ \sum_{k=1}^n \frac{X_k}{n} \longrightarrow \mathbb{E}[X] \,.$$

To be more precise, as $n \to \infty$, we have that the mean becomes a random variable that's distributed normally,
$$ \sum_{k=1}^n \frac{X_k}{n} \sim N\left(\mathbb{E}[X], \frac{\mbox{Var}[X]}{n} \right) .$$

Let's see that in action.  We'll plot very different distributions:

```python
# A variety of distributions, when averaged, are normal

dists = (
    sp.stats.uniform(),
    sp.stats.beta(a=2., b=4.),
    sp.stats.expon(),
    sp.stats.chi2(df=10),
    sp.stats.gamma(a=1.99),
    sp.stats.norm(),
)

N=2000
M=100

plt.figure(figsize=(8, 11))
for k, dist in enumerate(dists):
    ax = plt.subplot(6,2,2*k+1)
    plot_hist_dist(dist.rvs(size=N), dist, title=dist.dist.name + ' (original)', ax=ax)

    ax = plt.subplot(6,2,2*k+2)
    m, v = dist.stats()
    rvs = dist.rvs(size=(N,M)).mean(axis=1)
    plot_hist_dist(rvs, sp.stats.norm(loc=m, scale=np.sqrt(v / M)),
                   title=dist.dist.name + ' (central limit)', ax=ax)

plt.tight_layout()
```

## Beta Distribution

The Beta distribution is a distribution on $[0,1]$ that is very closely rated to the binomial distribution.  It has a PDF given by

$$ p(x) = \frac{x^{\alpha-1} (1-x)^{\beta - 1}}{\mbox{B}(\alpha, \beta)} $$

where $\mbox{B}(\alpha, \beta)$ is the beta function, which is just used as a normalizing constant to make $p(x)$ a probability distribution.  We'll denote this distribution $B(\alpha, \beta)$.

**Examples:**
- If $\alpha = 1$ and $\beta = 1$, this is just the uniform distribution.

**Stats:**
- The mean is $\mathbb{E}[X] = \frac{\alpha}{\alpha + \beta}$
- The variance is $\mbox{Var}[X] = \frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)}$.

```python
# Beta distribution, two parameters

a_slider = IntSlider(value=1, min=1, max=5, description='a')
b_slider = IntSlider(value=1, min=1, max=5, description='b')
interact(
    interact_dist(sp.stats.beta, xlim=[0, 1], ylim=[0, 3]),
    a=a_slider, b=b_slider
);
```

## Bayes' Theorem

Bayes' Theorem is a simple rule about conditional probabilities that has profound consequences.

$$ \mathbb{P}[X=x \mid Y=y] = \frac{\mathbb{P}[Y=y \mid X=x] \cdot \mathbb{P}[X=x]}{\mathbb{P}[Y=y]} $$

Mathematically, it's simply derived from the fact that
$$ \mathbb{P}[X=x\mid Y=y] \cdot \mathbb{P}[Y=y] = \mathbb{P}[X=x, Y=y] = \mathbb{P}[Y=y, X=x] = \mathbb{P}[Y=y \mid X=x] \cdot \mathbb{P}[X=x] \,.$$

While it seems trivial, it tells us what we know about one variable ($X$) given information about another ($Y$).

**Example:** Assume we have  biased Bernoulli $X$ that takes on $1$ with probability $P$.  We don't know $P$; we can call that a random variable as well, and let's assume it's beta distributed with parameters $\alpha$ and $\beta$ (denoted $B(\alpha, \beta)$).  By observing flips of $X$, we can intuitively infer the value of $P$.

Using Bayes' Theorem, we have that the new distribution of
$$
\begin{align}
\mathbb{P}[P=p \mid X=1] \propto\,\, & \mathbb{P}[X=1 \mid P=p] \cdot \mathbb{P}[P=p] \\
 \propto\,\, & p \cdot p^{\alpha-1} (1-p)^{\beta-1} \\
 \propto\,\, & p^{\alpha} (1-p)^{\beta-1}
\end{align}
$$
It's not hard to show that
$$ \mathbb{P}[P=p \mid X=1] = B(\alpha+1, \beta)\,. $$
Similarly,
$$ \mathbb{P}[P=p \mid X=0] = B(\alpha, \beta+1)\,. $$

Stated differently, if we have $P$ distributed as $B(\alpha, \beta)$ (the **prior**), and if we observe $X=1$, we update our estimate of the distribution of $P$ (the **posterior**) to $B(\alpha+1, \beta)$, and if we observe $X=0$, we update the posterior to $B(\alpha, \beta+1)$.  The fact that the Bayes' Theorem equations can be solved in closed form makes the Bernoulli and Beta distributions **Conjugate Priors**.

**Example:** We serve ads to individuals, and our prior belief that they will click on an ad is $P$ distributed $B(\alpha, \beta)$.  If we see them click on the ad, then our posterior for $P$ is $B(\alpha+1, \beta)$, and if they do not, then our posterior is $B(\alpha, \beta+1)$.

Continuing on, if we show another ad to the same individual, we can get this process to update again.  So if they click on 2 of the 10 ads we show them, then our posterior $P$ is $B(\alpha+2, \beta+8)$.

In fact, the canonical interpretation of $B(\alpha, \beta)$ is our belief about the odds of some $X$ being successful if we have seen $\alpha$ 1s and $\beta$ 0s.

Below is Bayes' Theorem in action:

```python
# Beta and binomial are conjugate priors

def binomial_bayes(a_0, b_0, a_1, b_1):
    prior = sp.stats.beta(a=a_0, b=b_0)
    posterior = sp.stats.beta(a=a_0+a_1, b=b_0+b_1)

    prior_rvs = prior.rvs(size=10000)
    variates = sp.stats.binom(n=a_1+b_1, p=prior_rvs).rvs()
    posterior_rvs = prior_rvs[variates == a_1]

    ax1 = plt.subplot(2,1,1)
    plot_hist_dist(prior_rvs, prior, title="Prior Beta(a={a}, b={b})".format(**prior.kwds), ax=ax1)
    ax2 = plt.subplot(2,1,2, sharex=ax1)
    plot_hist_dist(posterior_rvs, posterior, title="Posterior Beta(a={a}, b={b})".format(**posterior.kwds), ax=ax2)

    plt.tight_layout()

binomial_bayes(a_0=2, b_0=2, a_1=6, b_1=3)
```

Continuing with the previous example,  we expect these three facts to be true intuitively:
- If $X=1$, we expect our estimate of $P$ to go up.
- If $X=0$, we expect our estimate of $P$ to go down.
- In either case, we expect that we have learned more information about $P$ and we expect the uncertainty of $P$ to decrease.

We can see these 3 facts in the plots of $\beta$ below:

```python
# Intuition of beta distribution in terms of the Bernoulli

dists = (
    sp.stats.beta(a=2, b=2),
    sp.stats.beta(a=2, b=6),
    sp.stats.beta(a=6, b=2),
    sp.stats.beta(a=6, b=6),
)

N=2000
M=100

plt.figure()
for k, dist in enumerate(dists):
    ax = plt.subplot(2,2,k+1)
    plot_hist_dist(
        dist.rvs(size=N),
        dist,
        title="B(a={a},b={b})".format(**dist.kwds),
        ax=ax
    )
    ax.set_xlim([0, 1])
plt.suptitle('Distributions')
plt.tight_layout()
```

*Copyright © 2019 [Pragmatic Institute](https://www.pragmaticmarketing.com/data-science). This content is licensed solely for personal use. Redistribution or publication of this material is strictly prohibited.*


*Sebastian Raschka*
last modified: 03/31/2014

I am really looking forward to your comments and suggestions to improve and extend this tutorial! Just send me a quick note
via Twitter: [@rasbt](https://twitter.com/rasbt)
or Email: [bluewoodtree@gmail.com](mailto:bluewoodtree@gmail.com)

### Problem Category
- Statistical Pattern Recognition
- Supervised Learning
- Parametric Learning
- Bayes Decision Theory
- Univariate data
- 2-class problem
- equal variances
- equal priors
- Gaussian model (2 parameters)
- No Risk function

# Sections

• Given information

• Deriving the decision boundary

• Plotting the class conditional densities, posterior probabilities, and decision boundary

• Classifying some random example data

• Calculating the empirical error rate

## Given information:

[back to top]

####model: continuous univariate normal (Gaussian) model for the class-conditional densities

$ p(x | \omega_j) \sim N(\mu|\sigma^2) $

$ p(x | \omega_j) \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu}{\sigma}\bigg)^2 \bigg] } $

####Prior probabilities:

$ P(\omega_1) = P(\omega_1) = 0.5 $

#### Variances of the sample distributions

$ \sigma_1^2 = \sigma_2^2 = 1 $

#### Means of the sample distributions

$ \mu_1 = 4, \quad \mu_2 = 10 $

## Deriving the decision boundary
[back to top]

### Bayes' Rule:

$ P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)} $

###Bayes' Decision Rule:

Decide $ \omega_1 $ if $ P(\omega_1|x) > P(\omega_2|x) $ else decide $ \omega_2 $.

\begin{equation}
\Rightarrow \frac{p(x|\omega_1) * P(\omega_1)}{p(x)} > \frac{p(x|\omega_2) * P(\omega_2)}{p(x)}
\end{equation}

We can drop $ p(x) $ since it is just a scale factor.

$ \Rightarrow P(x|\omega_1) * P(\omega_1) > p(x|\omega_2) * P(\omega_2) $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{P(\omega_2)}{P(\omega_1)} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{0.5}{0.5} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > 1 $

$ \Rightarrow \frac{1}{\sqrt{2\pi\sigma_1^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2 \bigg] } > \frac{1}{\sqrt{2\pi\sigma_2^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \bigg] } $

Since we have equal variances, we can drop the first term completely.

$ Rightarrow \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2 \bigg] } >  \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \bigg] } \quad\quad \bigg| \;ln, \quad \mu_1 = 4, \quad \mu_2 = 10, \quad \sigma=1 $

$ \Rightarrow -\frac{1}{2} (x-4)^2  > -\frac{1}{2} (x-10)^2  \quad \bigg| \; \times(-2) $

$ \Rightarrow (x-4)^2  < (x-10)^2 $

$ \Rightarrow x^2 - 8x + 16 < x^2 - 20x + 100 $

$ \Rightarrow 12x < 84 $

$ \Rightarrow x < 7 $

## Plotting the class conditional densities, posterior probabilities, and decision boundary

[back to top]

```
%pylab inline

import numpy as np
from matplotlib import pyplot as plt

def pdf(x, mu, sigma):
    """
    Calculates the normal distribution's probability density
    function (PDF).

    """
    term1 = 1.0 / ( math.sqrt(2*np.pi) * sigma )
    term2 = np.exp( -0.5 * ( (x-mu)/sigma )**2 )
    return term1 * term2

# generating some sample data
x = np.arange(0, 100, 0.05)

# probability density functions
pdf1 = pdf(x, mu=4, sigma=1)
pdf2 = pdf(x, mu=10, sigma=1)

# Class conditional densities (likelihoods)
plt.plot(x, pdf1)
plt.plot(x, pdf2)
plt.title('Class conditional densities (likelihoods)')
plt.ylabel('p(x)')
plt.xlabel('random variable x')
plt.legend(['p(x|w_1) ~ N(4,1)', 'p(x|w_2) ~ N(10,1)'], loc='upper right')
plt.ylim([0,0.5])
plt.xlim([0,20])
plt.show()
```

```
def posterior(likelihood, prior):
    """
    Calculates the posterior probability (after Bayes Rule) without
    the scale factor p(x) (=evidence).

    """
    return likelihood * prior

# probability density functions
posterior1 = posterior(pdf(x, mu=4, sigma=1), 0.5)
posterior2 = posterior(pdf(x, mu=10, sigma=1), 0.5)

# Class conditional densities (likelihoods)
plt.plot(x, posterior1)
plt.plot(x, posterior2)
plt.title('Posterior Probabilities w. Decision Boundary')
plt.ylabel('P(w)')
plt.xlabel('random variable x')
plt.legend(['P(w_1|x)', 'p(w_2|X)'], loc='upper right')
plt.ylim([0,0.5])
plt.xlim([0,20])
plt.axvline(7, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(4, 0.3), xytext=(4, 0.3))
plt.annotate('R2', xy=(10, 0.3), xytext=(10, 0.3))
plt.show()
```

## Classifying some random example data

[back to top]

#### Note on generating univariate random data from a Normal Distribution

We can generate random samples drawn from a Normal distribution via the `np.random.randn()` function. Its default is a standard Normal distribution with $ \mu = 0 $ and $ \sigma^2 = 1 $. In order to draw random data from $ N(\mu, \sigma^2) $, we use
`sigma * np.random.randn(...) + mu`

```
# Parameters
mu_1 = 4
mu_2 = 10
sigma_1_sqr = 1
sigma_2_sqr = 1

# Generating 10 random samples drawn from a Normal Distribution for class 1 & 2
x1_samples = sigma_1_sqr**0.5 * np.random.randn(10) + mu_1
x2_samples = sigma_1_sqr**0.5 * np.random.randn(10) + mu_2
y = [0 for i in range(10)]

# Plotting sample data with a decision boundary

plt.scatter(x1_samples, y, marker='o', color='green', s=40, alpha=0.5)
plt.scatter(x2_samples, y, marker='^', color='blue', s=40, alpha=0.5)
plt.title('Classifying random example data from 2 classes')
plt.ylabel('P(x)')
plt.xlabel('random variable x')
plt.legend(['w_1', 'w_2'], loc='upper right')
plt.ylim([-0.1,0.1])
plt.xlim([0,20])
plt.axvline(7, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(4, 0.05), xytext=(4, 0.05))
plt.annotate('R2', xy=(10, 0.05), xytext=(10, 0.05))
plt.show()
```

## Calculating the empirical error rate

[back to top]

```
w1_as_w2, w2_as_w1 = 0, 0
for x1,x2 in zip(x1_samples, x2_samples):
    if x1 >= 7:
        w1_as_w2 += 1
    if x2 < 7:
        w2_as_w1 += 1

emp_err =  (w1_as_w2 + w2_as_w1) / float(len(x1_samples) + len(x2_samples))

print('Empirical Error: {}%'.format(emp_err * 100))

```

    Empirical Error: 0.0%

```

```


## Questionário - Semana 2

**Nota**: utilize este texto como acessório para inserir as respostas no questionário 2.

Leia o texto a seguir para responder o questionário.

### Modelos epidemiológicos SEIR

Modelos matemáticos epidemiológicos modernos funcionam com base em "estados" que são considerados como "compartimentos". Por exemplo, o modelo SEIR considera que uma população, quando ameaçada por uma epidemia, pode ser dividida em quatro compartimentos:

- S, das pessoas *suscetíveis* à infecção;
- E, das pessoas *expostas*, que embora hospedeiras, não espalham a doença pelo fato de o vírus ainda estar em incubação;
- I, das pessoas *infectadas* e
- R, das pessoas *recuperadas*

O número de pessoas $N$ em cada "compartimento" muda com o tempo. Logo, a dinâmica de mudança de estados ocorre por transições. Em cada instante de tempo, um fluxo de transição da forma
S -> E -> I -> R pode ocorrer. Isto é, pessoas suscetíveis podem contrair o vírus, em seguida tornar-se expostas, serem infectadas e, por sua vez, se recuperarem. Portanto, supondo que não haja mortalidade, $N$ deve se conservar ao longo do tempo, mas S, E, I e R podem variar com o tempo. Matematicamente, se $t$ é um instante de tempo, então:

$$N(t) = S(t) + E(t) + I(t) + R(t)$$

Um modelo SEIR busca prever o número de pessoas em cada estado em um instante de tempo futuro $t_N$ (dia N) a partir de um instante de tempo inicial $t_0$ (dia 0).

Considere que $a(t)$ seja o número de pessoas que uma pessoa infecta considerado para o dia $t$ – diante de um quadro epidêmico, desejamos que este valor seja o mais próximo de zero possível. Este valor pode se alterar para mais ou para menos mediante o relaxamento ou restrição de medidas de distanciamento social. A função matemática

$$a(t) = a_N + \dfrac{a_0 - a_N}{1 + e^{-k(L-t)}},$$

pode ser usada como um modelo para o cálculo de $a$ levando em consideração medidas drásticas de "lockdown". Na equação:

- $a_0$ é o valor de $a$ no dia 0
- $a_N$ é o valor de $a$ no dia N
- $L$ é o dia entre o dia 0 e o dia N em que o "lockdown" é decretado.
- $k$ é uma constante de ajuste no intervalo real $(0,1]$ que aumenta a velocidade de declínio de $a$ à medida que se aproxima de $1$.

A figura a seguir mostra o gráfico de $a(t)$ para $k = 0.2$, $k = 0.4$ e $k = 0.9$ em um intervalo de 100 dias, considerando $a_0 = 7$ e $a_{100} = 0.5$. O lockdown foi determinado no dia 40. Como se vê, quanto maior o valor de $k$, mais brusca é a queda do poder de contágio de uma pessoa.

 </img>

## Questão 1

Construa um modelo simbólico para $a(t)$ e o atribua à variavel `a`. Qual(is) alternativa(s) seguinte(s) pode(m) ser uma resposta esperada por `type(a)`, independentemente da construção?

**Ver opções no Moodle**

## Questão 2

Considere as seguintes situações:

1) A cidade A decretou lockdown 30 dias após o dia inicial de observação (dia 0) do contágio epidêmico, assumindo que $a_0 = 5$, $a_{100} = 1$ e modelou a curva $a(t)$ com $k = 0.2$.

2) A cidade B decretou lockdown 40 dias após o dia inicial de observação (dia 0) do contágio epidêmico, assumindo que $a_0 = 5$, $a_{100} = 1$ e modelou a curva $a(t)$ com $k = 0.1$.

Sendo $P_1$ e $P_2$ os valores de $a(t)$ calculados pelos modelos das cidades A e B, respectivamente, para o 21o. dia após o início da análise, o que podemos afirmar?

**Ver opções no Moodle**

## Questão 3

Considere que a população de uma cidade acometida pela epidemia seja de 38.416 habitantes. No 16o. dia, a análise mostrou que o número de pessoas suscetíveis ao contágio era o triplo das que estavam já em exposição ao vírus. Estas, por sua vez, estavam em mesmo número que as infectadas no dia. As recuperadas contabilizavam exatamente 16.

O código abaixo, iniciado por um analista, buscou determinar o percentual de pessoas expostas em relação à população total da cidade no 16o. dia.

```python
from sympy.abc import N,S,E,I,R
from sympy import Eq, solveset
tot = Eq( S + E + I + R - N,0)
...
```

Ao completar este código, ou desenvolver o seu próprio, você pode concluir que este percentual é de:

**Ver opções no Moodle**

## Questão 4

Algumas funções anônimas foram programadas na tentativa de representar o termo $e^{-k(L-t)}$ que aparece no denominador da fração do modelo de $a(t)$. Assinale a alternativa cuja definição expressa corretamente o termo, tendo em vista que `exp` é a exponencial simbólica:

**Ver opções no Moodle**

## Questão 5

Considere o seguinte excerto de código:

```python
from sympy.abc import S,E,I,R
SEIR = S*E*I*R
```

Podemos concluir que

$$\text{sen}( SEIR + SEIR^{-1} + 1/SEIR^{-1}) = \text{sen}(x + y),$$

onde

**Ver opções no Moodle**


# Exercise Session 2: Convolution, Filtering and Gradients

The goal of this exercise is to:
* Understand and apply a convolutional filter to an image
* Compare the computational complexity of separable and non-separable filters
* Compute image gradients and apply them to real-world images

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import cv2
import time
import scipy.signal as conv

plt.rcParams['figure.figsize'] = (10, 10)
plt.rcParams['image.cmap'] = 'gray'
```

## Exercise 1: Applying Convolutional Filters

Assume we are given a gray-scale image $I[x, y]$, of size $W \times H$, such that $0 \leq x \leq W-1$,
and $0 \leq y \leq H-1$. We want to apply a filter $F[i, j]$ to image $I$. The filter $F$ is of size $(2N + 1) \times (2M + 1)$, such that $−N \leq i \leq N$, and $−M \leq j \leq M$.

The result can be computed as

\begin{align}
R[x, y] = (I ∗ F)[x, y] = \sum_{i=-N}^{N} \sum_{i=-M}^{M} I[x − i, y − j]~F[i, j]
\end{align}

* Implement a function ```R = applyImageFilter(I,F)``` that takes a
gray-scale image $I$ and a filter $F$ as inputs, and returns the result of the convolution
of the two.
Note: There are many python libraries providing convolution function to convolve an image with a filter,
but in this exercise you are requested to implement your own. This will help your understanding
of how convolutional filters work.
    -  To avoid numerical issues, make sure $I$ and $F$ are of type float. To understand
why, think what would happen if you add two unsigned 8-bit numbers when
computing the convolution, for example 240 and 80.
    -  What happens when computing $R[x, y]$ near the border of the image? What
would you propose to deal with this issue?

```python
def applyImageFilter(I, F, padding='same'):
    # First input parameter: I
    #     Input image. It should be a 2D matrix. According to the notation in the description, it has W rows and
    #     H columns.
    # Second input parameter: F
    #     Filter used for the convolution. It should be a 2D matrix. According to the notation in the decription,
    #     it has (2N+1) rows and (2M+1) columns.

    # If image is not of type float, convert it to float
    if not np.issubdtype(I.dtype,float):
        I = np.float64(I)

    # If the filter is not of type float, convert it to float
    if not np.issubdtype(F.dtype,float):
        F = np.float64(F)

    # Shape of Filter
    N_, M_ = F.shape

    # Check whether the dimensions of input are accurate, they should be odd
    if not N_%2:
        raise ValueError('Number of rows in the filter must be odd')

    if not M_%2:
        raise ValueError('Number of columns in the filter must be odd')

    # Compute the values for N,M which is used in the above description.
    N = np.int64((N_-1)/2)
    M = np.int64((M_-1)/2)

    # Shape of the input image
    W, H = I.shape

    # Initialize output matrix R
    # "same" padding mode: allows conv at boundary pixels, final output image size = input image size
    # default padding value = 0, initialize output matrix to the original input shape
    if padding == 'same':
        I = np.pad(I, ((N,N),(M,M)), 'constant', constant_values=0)
        R = np.zeros((W,H), dtype=np.float64)

    # "valid" padding mode: no conv for boundary pixels, output image size reduced to (W-2M)*(H-2N)
    elif padding == 'valid':
        R = np.zeros((W-2*M, H-2*N), dtype=np.float64)

    # if not any mode, raise ValueError
    else:
        raise ValueError("Please choose padding='same' or 'valid'.")

    # Output image size
    W_R, H_R = R.shape

    # Perform 2D convolution
    for x in range(W_R):
        for y in range(H_R):
            # calculate R[x,y]
            # iterate over N,M to get conv value
            for i in range(-N, N+1):
                for j in range(-M, M+1):
                    R[x,y] += I[x-i+N, y-j+M] * F[i+N,j+M]

    return R
```

```python
# Below code is to verify your implementation of the convolution
# we compare your output with the scipy.signal implementation of the convolution
# and the error should be zero

# Read the sample image
img = cv2.imread('images/res1.png',0)

# Generate a random filter F of size 3 x 3
F = np.random.randn(5,3)

# Perform convolution with the function you implemented
output_1 = applyImageFilter(img, F, padding='same')

# Perform convolution with scipy.signal implementation of convolution
output_2 = conv.convolve2d(img, F , mode='same')

# Error
print(f'Input shape: {img.shape}')
print(f'Output shape: {output_1.shape}')
print(f'Error: {np.sum(np.abs(output_1 - output_2))}')
# if the implmentation is accurate, the error will be 0
```

    Input shape: (390, 476)
    Output shape: (390, 476)
    Error: 0.0

## Exercise 2: Image Smoothing

Convolutional filters have many uses. A very common one is to smooth or soften an
image. A typical smoothing filter is the Gaussian Filter, which follows the equation of a
2D Gaussian probability distribution. It is an important preprocessing step typically to reduce the amount of noise.  When constructing a Gaussian filter we can specify the size and standard deviation (σ) of the underlying Gaussian function.

* To create a Gaussian filter use the function given below.

```python
def gaussian_filter(fSize, fSigma):
    x, y = np.mgrid[-fSize//2 + 1:fSize//2 + 1, -fSize//2 + 1:fSize//2 + 1]
    g = np.exp(-((x**2 + y**2)/(2.0*fSigma**2)))
    return g/g.sum()
```

* What are fSize and fSigma?
* How do fSize and fSigma affect the filter’s shape and size?
* Visualize different filters by giving different values to these parameters.
* If you are given fSigma, how would you choose fSize?

```python
# Here we visualize some combinations of fSize and fSigma
size_list = [3, 5, 7]
sigma_list = [1, 3, 6]

fig, axes = plt.subplots(len(size_list), len(sigma_list))
for pos in [(i,j) for i in range(len(size_list)) for j in range(len(sigma_list))]:
    axes[pos].imshow(gaussian_filter(size_list[pos[0]], sigma_list[pos[1]]), cmap='viridis')
```

**Answer to Questions**:
1. fSize is the width/length of the gaussian filter, fSigma is the standard deviation for the gaussian filter/kernel.
2. fSize controls the size(width, height) of the filter, fSigma controls how the elements in gaussian kernel variates.
3. A rule of thumb of choosing the filter size: $k = ceil(3\sigma)$ for k*k kernel, k is odd

* Apply Gaussian filtering to 'res1.png'
* To apply the filter to an image use ```cv2.GaussianBlur(img, fSize, fSigma)```
* Experiment with different values of fSigma. How does the amount of smoothing vary with this parameter?

```python
img = cv2.imread('images/res1.png',0)
plt.imshow(img)

# apply filters, try different values of Sigma
size_list = [3, 5, 7]
sigma_list = [1, 3, 6]

fig, axes = plt.subplots(len(size_list), len(sigma_list))
for pos in [(i,j) for i in range(len(size_list)) for j in range(len(sigma_list))]:
    axes[pos].imshow(cv2.GaussianBlur(img, [size_list[pos[0]],size_list[pos[0]]], sigma_list[pos[1]]))
```

As you saw in the first exercise of this session, applying the filter near the border of
the image is not a well-defined problem. It is possible to deal with this issue through
different approaches. Luckily, cv2.GaussianBlur implements different ways of dealing with border effects.
Here you will try them out and understand what each of them does.

*  Apply a Gaussian filter with fSigma = 10 and fSize = 25 to 'res1.png'
* Use borderType flag of cv2.GaussianBlur function to define what kind of border should be added around the image (cv2.BORDER_CONSTANT, cv2.BORDER_REFLECT, cv2.BORDER_REPLICATE).

```python
img = cv2.imread('images/res1.png',0)
# apply different border types for blur filters

fig, axes = plt.subplots(1, 3)
axes[0].imshow(cv2.GaussianBlur(img, [25,25], 10, borderType=cv2.BORDER_CONSTANT))
axes[1].imshow(cv2.GaussianBlur(img, [25,25], 10, borderType=cv2.BORDER_REFLECT))
axes[2].imshow(cv2.GaussianBlur(img, [25,25], 10, borderType=cv2.BORDER_REPLICATE))
```

## Exercise 3: Edge detector

Edge detection is one of the most important operations in Computer Vision. In this exercise we will investigate how it
can be performed and what information can be obtained from it.

One of the ways to detect edges is to compute image intensity gradients. They can be approximated by convolving image with Sobel filters. They consist of 2 $3 \times 3$ filters:

\begin{equation*}
S_x = \frac{1}{8}
\begin{bmatrix}
-1 & 0 & +1\\
-2 & 0 & +2\\
-1 & 0 & +1\\
\end{bmatrix}
\hspace{2cm}
S_y = \frac{1}{8}
\begin{bmatrix}
-1 & -2 & -1\\
0 & 0 & 0\\
+1 & +2 & +1\\
\end{bmatrix}
\end{equation*}

where $S_x$ computes the partial derivative of the image in the horizontal direction, while $S_y$ does it in the vertical direction.

* Compute the image derivatives in x- and y-directions using Sobel filters. You can use for that function ```python cv2.Sobel()```

```python
img_coins = cv2.imread('images/coins.png',0)

### Compute gradient in x-direction
grad_x = cv2.Sobel(img_coins, ddepth=cv2.CV_64F, dx=1, dy=0, ksize=3)

### Compute gradient in y-direction
grad_y = cv2.Sobel(img_coins, ddepth=cv2.CV_64F, dx=0, dy=1, ksize=3)
```

* Visualize the image gradients. How can you explain the differences between the two results?

```python
fig = plt.figure()
plt.subplot(1,2,1)
grady = plt.imshow(grad_x,cmap="jet")
plt.title("Gradient x")
plt.colorbar(fraction=0.046, pad=0.04)
plt.subplot(1,2,2)
gradx = plt.imshow(grad_y,cmap="jet")
plt.title("Gradient y")
plt.colorbar(fraction=0.046, pad=0.04)
plt.tight_layout()
```

Mathematically, the goal of the filters $S_x$ and $S_y$ is to approximate the derivatives of the image with respect to
the horizontal and vertical directions respectively, such that
\begin{equation*}
\nabla I_x(x,y) = (I*S_x)[x,y]
\quad\mathrm{and}\quad
\nabla I_y(x,y) = (I*S_y)[x,y]
\end{equation*}

Therefore, the gradient of an image at each point is a 2D vector
\begin{equation*}
\nabla I =
\begin{bmatrix}
\nabla I_x\\
\nabla I_y
\end{bmatrix}
\end{equation*}

This vector can be computed for every pixel. Its magnitude and phase can be computed as
\begin{equation*}
|| \nabla I || = \sqrt{(\nabla I_x)^2 + (\nabla I_y)^2} \\
\angle \nabla I = atan2(\nabla I_x, \nabla I_y)
\end{equation*}

$\it{Note:}$ we use atan2() instead of atan() to be able to determine the right quadrant of the phase.

* Using the previously computed image gradients and the above formula compute the gradient magnitude and gradient phase.

```python
### Compute gradient magnitude
grad_mag = np.sqrt(np.power(grad_x, 2) + np.power(grad_y, 2))

### Compute gradient phase
grad_phase = np.arctan2(grad_x, grad_y)
```

* Visualize the gradient magnitude and phase images along with the original image. Where does the gradient have high
magnitude? How does the phase change along the coin border?

```python
fig = plt.figure()
plt.subplot(1,3,1)
plt.imshow(img_coins)
plt.subplot(1,3,2)
grady = plt.imshow(grad_mag,cmap="jet")
plt.title("Gradient magnitude")
plt.colorbar(fraction=0.046, pad=0.04)
plt.subplot(1,3,3)
gradx = plt.imshow(grad_phase)
plt.title("Gradient phase")
plt.colorbar(fraction=0.046, pad=0.04)
plt.tight_layout()
```

The gradient has got the highest magnitude at the borders of the coins. For that reason it can be considered a good
proxy for detecting edges. The phase of the gradient changes smoothly along the circular coin border along with the
normal of the boundary.

## Exercise 4: Separable Filters

As seen in class, certain types of 2D filters can be thought of as the composition of two
1-dimensional filters. These are called Separable Filters, and can be computed more
efficiently than those who are non-separable.

For example, the Sobel filter $S_x$ can be decomposed as

\begin{equation*}
\begin{bmatrix}
-1 & 0 & +1\\
-2 & 0 & +2\\
-1 & 0 & +1\\
\end{bmatrix} =
\begin{bmatrix}
+1 \\
+2 \\
+1 \\
\end{bmatrix}
\begin{bmatrix}
-1 & 0 & +1
\end{bmatrix}
\end{equation*}

which means that the convolution of I with the 2D filter $S_x$ can be simplified to two
convolutions with 1D filters,

\begin{equation*}
I * S_x =
\frac{1}{8} I *
\begin{bmatrix}
-1 & 0 & +1\\
-2 & 0 & +2\\
-1 & 0 & +1\\
\end{bmatrix}
\end{equation*}

\begin{equation*}
=
\frac{1}{8}\Bigg(I *
\begin{bmatrix}
+1\\
+2\\
+1\\
\end{bmatrix} \Bigg) *
\begin{bmatrix}
-1 & 0 & +1
\end{bmatrix}
\end{equation*}

For the comparison of the computational complexity of separable and non-separable
filters, check out the lecture notes.

* We will compare the performance of separable and non-separable filters.
    -  Create a separable filter (for example a Gaussian filter, you can use  ```gaussian_filter``` function given below)
    -  Use ```decomposeSeparableFilter``` to decompose the seperable filter to two, 1D filters.
    -  Implement ```applyImageSepFilter``` function. You can use ```applyImageFilter``` function implemented in Applying Convolutional Filters (Hint: two lines of code)
    -  Perform convolution on ```img``` matrix and measure the time for ```decomposeSeparableFilter``` and ```applyImageSepFilter```.
    -  Repeat the test with several increasing filter sizes.
    -  measure the elapsed time for each filtering. To get the start and end time, use time.time()
    -  Plot the results in a graph (elapsed time) vs (filter size).
    -  What is your conclusion ? Is filtering with separable filters always faster than with non-separable ones?

```python
def decomposeSeparableFilter(F):
    h = [1]
    s = len(F)
    for i in range(1,s):
            h.append(np.sum(F[:,i])/(np.sum(F[:,0])))
    h = np.asmatrix(np.array(h))
    v = np.asmatrix(F[:,0]).transpose()
    return v, h

# implement the convolution with two 1D filters and return the output
def applyImageSepFilter(I, F_v, F_h):
    I = applyImageFilter(I, F_h, padding='same')
    output = applyImageFilter(I, F_v, padding='same')
    return output
```

```python
# Sample image
img = cv2.imread('images/res1.png',0)
img = img[150:200,150:200]
# We crop the image, becasue larger images take considerably longer time to
# perform convolution with large filters

# Filter sizes to try out.
# Since we consider filter with odd heigh and width,
# we start from 3 and increment it by 2 till 29.
filter_sizes = range(3,30,2)
```

```python
# ******************************************
# Perform the comparison here
# Hint: Iterate through the filter_sizes and in each iteration perform convolution with the 2D and 1D filters
#         using applyImageFilter and applyImageSepFilter functions respetively. Measure the time for each execution.
#         Store the execution times and plot them at the end
# Note: Performing convolutions with larger filters could take longer time (sometimes more than a minute)
# ******************************************
time_records = []

for size in filter_sizes:
    F = gaussian_filter(fSize=size, fSigma=1)
    F_v, F_h = decomposeSeparableFilter(F)

    # measure time used for 2D filter conv
    start_2D = time.time()
    output_2D = applyImageFilter(img, F, padding='same')
    end_2D = time.time()
    time_2D = end_2D - start_2D

    start_1D = time.time()
    output_1D = applyImageSepFilter(img, F_v, F_h)
    end_1D = time.time()
    time_1D = end_1D - start_1D

    # check if the error of two conv result is within an accepatable range
    print("Different of 2D and 1D separate conv result: {}".format(np.sum(np.abs(output_2D-output_1D))))

    time_records.append([time_2D, time_1D])
```

    Different of 2D and 1D separate conv result: 2.439293211864424e-11
    Different of 2D and 1D separate conv result: 3.54631879417866e-11
    Different of 2D and 1D separate conv result: 4.461497837837669e-11
    Different of 2D and 1D separate conv result: 7.858602657506708e-11
    Different of 2D and 1D separate conv result: 7.577938276881468e-11
    Different of 2D and 1D separate conv result: 7.683098601773963e-11
    Different of 2D and 1D separate conv result: 1.0412293249828508e-10
    Different of 2D and 1D separate conv result: 1.3211831628723303e-10
    Different of 2D and 1D separate conv result: 1.0696510344132548e-10
    Different of 2D and 1D separate conv result: 1.552464823362243e-10
    Different of 2D and 1D separate conv result: 1.4057377484277822e-10
    Different of 2D and 1D separate conv result: 1.0876277656279854e-10
    Different of 2D and 1D separate conv result: 1.5565859712296515e-10
    Different of 2D and 1D separate conv result: 1.3151435496183694e-10

```python
time_records = np.array(time_records)

plt.figure(figsize=(8, 4))
plt.plot(filter_sizes, time_records[:, 0], label="2D Filter Conv")
plt.plot(filter_sizes, time_records[:, 1], label="1D Filter Conv")
plt.xlabel("Filter Size")
plt.ylabel("Time Elapsed 10^(-11)s")
plt.legend()
```

```python

```

ExerciseSession2/Convolution, Filtering and Gradients.ipynb


```python
import sympy as sp
import Resources.script as rt
```

## Zadaca 9

```python
r, beta, gamma = sp.symbols('r, beta, gamma')
T = rt.hrotation3('z', gamma) * rt.hrotation3('y', beta) * rt.htranslation3(z=r)
T
```

$\displaystyle \left[\begin{matrix}\cos{\left(\beta \right)} \cos{\left(\gamma \right)} & - \sin{\left(\gamma \right)} & \sin{\left(\beta \right)} \cos{\left(\gamma \right)} & r \sin{\left(\beta \right)} \cos{\left(\gamma \right)}\\\sin{\left(\gamma \right)} \cos{\left(\beta \right)} & \cos{\left(\gamma \right)} & \sin{\left(\beta \right)} \sin{\left(\gamma \right)} & r \sin{\left(\beta \right)} \sin{\left(\gamma \right)}\\- \sin{\left(\beta \right)} & 0 & \cos{\left(\beta \right)} & r \cos{\left(\beta \right)}\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
pos = T[:3,3]
jacob = pos.jacobian([r,beta,gamma])
jacob
```

$\displaystyle \left[\begin{matrix}\sin{\left(\beta \right)} \cos{\left(\gamma \right)} & r \cos{\left(\beta \right)} \cos{\left(\gamma \right)} & - r \sin{\left(\beta \right)} \sin{\left(\gamma \right)}\\\sin{\left(\beta \right)} \sin{\left(\gamma \right)} & r \sin{\left(\gamma \right)} \cos{\left(\beta \right)} & r \sin{\left(\beta \right)} \cos{\left(\gamma \right)}\\\cos{\left(\beta \right)} & - r \sin{\left(\beta \right)} & 0\end{matrix}\right]$

```python
detJ = jacob.det().simplify()
detJ
```

$\displaystyle r^{2} \sin{\left(\beta \right)}$

```python
interval = sp.Interval(-sp.pi, sp.pi, left_open=True)
sp.solveset(detJ,beta,interval)
```

$\displaystyle \left\{0, \pi\right\}$

## Zadaca 10

```python
d1, d2, theta3, a3 = sp.symbols('d1, d2, theta3, a3')
robot = rt.SerialLinkRobot()
robot.add_prismatic_joint(0, d1, 0, -sp.pi/2)
robot.add_prismatic_joint(0, d2, 0, sp.pi/2)
robot.add_revolute_joint(theta3, 0, a3, 0)
robot.add_subs([(a3, 1)])
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
robot.jacobian()
```

$\displaystyle \left[\begin{matrix}0 & 0 & - a_{3} \sin{\left(\theta_{3} \right)}\\0 & 1 & a_{3} \cos{\left(\theta_{3} \right)}\\1 & 0 & 0\\0 & 0 & 0\\0 & 0 & 0\\0 & 0 & 1\end{matrix}\right]$

## Zadaca 18

```python
d1, theta2, l1, l2 = sp.symbols('d1, theta2, l1, l2')
robot = rt.SerialLinkRobot()
robot.add_prismatic_joint(0, d1, l1, sp.pi/2)
robot.add_revolute_joint(theta2, 0, l2, 0)
robot.add_subs([(l1, 1), (l2, 1)])
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
Jv = robot.linear_jacobian()
Jv
```

$\displaystyle \left[\begin{matrix}0 & - l_{2} \sin{\left(\theta_{2} \right)}\\0 & 0\\1 & l_{2} \cos{\left(\theta_{2} \right)}\end{matrix}\right]$

```python
Jv2 = sp.Matrix.vstack(Jv[:1, :], Jv[2:, :])
Jv2
```

$\displaystyle \left[\begin{matrix}0 & - l_{2} \sin{\left(\theta_{2} \right)}\\1 & l_{2} \cos{\left(\theta_{2} \right)}\end{matrix}\right]$

```python
Jv2.det()
```

$\displaystyle l_{2} \sin{\left(\theta_{2} \right)}$

```python
interval = sp.Interval(-sp.pi, sp.pi, left_open=True)
solution = sp.solveset(Jv2.det(), theta2, interval)
solution
```

$\displaystyle \left\{0, \pi\right\}$

```python
theta1, theta2, theta3, l1, l2, l3 = sp.symbols('theta1, theta2, theta3, l1, l2, l3')
robot = rt.SerialLinkRobot()
robot.add_revolute_joint(theta1, 0, l1, 0)
robot.add_revolute_joint(theta2, 0, l2, 0)
robot.add_revolute_joint(theta3, 0, l3, 0)
robot.add_subs([(l1, 1), (l2, 1), (l3, 1)])
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
J = robot.jacobian()
J
```

$\displaystyle \left[\begin{matrix}- l_{1} \sin{\left(\theta_{1} \right)} - l_{2} \sin{\left(\theta_{1} + \theta_{2} \right)} - l_{3} \sin{\left(\theta_{1} + \theta_{2} + \theta_{3} \right)} & - l_{2} \sin{\left(\theta_{1} + \theta_{2} \right)} - l_{3} \sin{\left(\theta_{1} + \theta_{2} + \theta_{3} \right)} & - l_{3} \sin{\left(\theta_{1} + \theta_{2} + \theta_{3} \right)}\\l_{1} \cos{\left(\theta_{1} \right)} + l_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} + l_{3} \cos{\left(\theta_{1} + \theta_{2} + \theta_{3} \right)} & l_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} + l_{3} \cos{\left(\theta_{1} + \theta_{2} + \theta_{3} \right)} & l_{3} \cos{\left(\theta_{1} + \theta_{2} + \theta_{3} \right)}\\0 & 0 & 0\\0 & 0 & 0\\0 & 0 & 0\\1 & 1 & 1\end{matrix}\right]$

```python
det_J = sp.det().simplify()
det_J
```

$\displaystyle 0$

## Zadaca 19

```python
theta1, theta2, d3, theta4, a1, a2, d4 = sp.symbols('theta1, theta2, d3, theta4, a1, a2, d4')
robot = rt.SerialLinkRobot()
robot.add_revolute_joint(theta1, 0, a1, 0)
robot.add_revolute_joint(theta2, 0, a2, sp.pi)
robot.add_prismatic_joint(0, d3, 0, 0)
robot.add_revolute_joint(theta4, d4, 0, 0)
robot.add_subs([(a1, 1), (a2, 1), (d4, 1)])
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
J = robot.jacobian()
J
```

$\displaystyle \left[\begin{matrix}- a_{1} \sin{\left(\theta_{1} \right)} - a_{2} \sin{\left(\theta_{1} + \theta_{2} \right)} & - a_{2} \sin{\left(\theta_{1} + \theta_{2} \right)} & 0 & 0\\a_{1} \cos{\left(\theta_{1} \right)} + a_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} & a_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} & 0 & 0\\0 & 0 & -1 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\1 & 1 & 0 & -1\end{matrix}\right]$

```python
J4 = sp.Matrix.vstack(J[:3, :], J[5:, :])
J4
```

$\displaystyle \left[\begin{matrix}- a_{1} \sin{\left(\theta_{1} \right)} - a_{2} \sin{\left(\theta_{1} + \theta_{2} \right)} & - a_{2} \sin{\left(\theta_{1} + \theta_{2} \right)} & 0 & 0\\a_{1} \cos{\left(\theta_{1} \right)} + a_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} & a_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} & 0 & 0\\0 & 0 & -1 & 0\\1 & 1 & 0 & -1\end{matrix}\right]$

```python
det_J4 = sp.det(J[:2, :2]).simplify()
det_J4
```

$\displaystyle a_{1} a_{2} \sin{\left(\theta_{2} \right)}$

```python
interval = sp.Interval(-sp.pi, sp.pi, left_open=True)
solution = sp.solveset(det_J4, theta2, interval)
solution
```

$\displaystyle \left\{0, \pi\right\}$

```python

```

## Zadaca 20

```python
d1, theta2, theta3, d4, l2, l3 = sp.symbols('d1, theta2, theta3, d4, l2, l3')
robot = rt.SerialLinkRobot()
robot.add_prismatic_joint(0, d1, 0, 0)
robot.add_revolute_joint(theta2, 0, l2, 0)
robot.add_revolute_joint(theta3, 0, l3, sp.pi)
robot.add_prismatic_joint(0, d4, 0, 0)
robot.add_subs([(l2, 2), (l3, 1)])
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
j = robot.jacobian()
j
```

$\displaystyle \left[\begin{matrix}0 & - l_{2} \sin{\left(\theta_{2} \right)} - l_{3} \sin{\left(\theta_{2} + \theta_{3} \right)} & - l_{3} \sin{\left(\theta_{2} + \theta_{3} \right)} & 0\\0 & l_{2} \cos{\left(\theta_{2} \right)} + l_{3} \cos{\left(\theta_{2} + \theta_{3} \right)} & l_{3} \cos{\left(\theta_{2} + \theta_{3} \right)} & 0\\1 & 0 & 0 & -1\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 1 & 1 & 0\end{matrix}\right]$

```python
detj = sp.simplify(j[:2,1:3].det())
detj
```

$\displaystyle l_{2} l_{3} \sin{\left(\theta_{3} \right)}$

```python
interval = sp.Interval(-sp.pi, sp.pi, left_open=True)
solution = sp.solveset(detj, theta3, interval)
solution
```

$\displaystyle \left\{0, \pi\right\}$

## Zadaca 23

```python
theta1, theta2, d1, d2, d3 = sp.symbols('theta1, theta2, d1, d2, d3')
robot = rt.SerialLinkRobot()
robot.add_revolute_joint(theta1, d1, 0, sp.pi/2)
robot.add_revolute_joint(theta2, 0, d2, sp.pi/2)
robot.add_prismatic_joint(0, d3, 0, 0)
robot.add_subs([(d1, 1), (d2, 1)])
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
T03 = robot.get_dh_joint_to_joint(0,3)
T03
```

$\displaystyle \left[\begin{matrix}\cos{\left(\theta_{1} \right)} \cos{\left(\theta_{2} \right)} & \sin{\left(\theta_{1} \right)} & \sin{\left(\theta_{2} \right)} \cos{\left(\theta_{1} \right)} & \left(d_{2} \cos{\left(\theta_{2} \right)} + d_{3} \sin{\left(\theta_{2} \right)}\right) \cos{\left(\theta_{1} \right)}\\\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} \right)} & - \cos{\left(\theta_{1} \right)} & \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{2} \right)} & \left(d_{2} \cos{\left(\theta_{2} \right)} + d_{3} \sin{\left(\theta_{2} \right)}\right) \sin{\left(\theta_{1} \right)}\\\sin{\left(\theta_{2} \right)} & 0 & - \cos{\left(\theta_{2} \right)} & d_{1} + d_{2} \sin{\left(\theta_{2} \right)} - d_{3} \cos{\left(\theta_{2} \right)}\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
J = robot.jacobian()
J = J.subs([(theta1,0),(theta2,0),(d3,5),(d2,1)])
poc = sp.Matrix([0.1, -0.1, 0.05])
Tdelta = J*poc
Tdelta
```

$\displaystyle \left[\begin{matrix}-0.5\\0.1\\-0.15\\0\\0.1\\0.1\end{matrix}\right]$

```python
delta = rt.hdelta3(dRx=0, dRy=0.1, dRz=0.1, dx=-0.5, dy=0, dz=-0.05)
delta
```

$\displaystyle \left[\begin{matrix}0 & -0.1 & 0.1 & -0.5\\0.1 & 0 & 0 & 0\\-0.1 & 0 & 0 & -0.05\\0 & 0 & 0 & 0\end{matrix}\right]$

```python
theta1, theta2, theta3, l1, l2, l3 = sp.symbols('theta1, theta2, theta3, l1, l2, l3')
robot = rt.SerialLinkRobot()
robot.add_revolute_joint(theta1, 0, l1, sp.pi/2)
robot.add_revolute_joint(theta2, 0, l2, 0)
robot.add_revolute_joint(theta3, 0, l3, 0)
robot.add_subs([(l1, 1), (l2, 1), (l3, 2)])
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
J = robot.jacobian()
J
```

$\displaystyle \left[\begin{matrix}- \left(l_{1} + l_{2} \cos{\left(\theta_{2} \right)} + l_{3} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \sin{\left(\theta_{1} \right)} & - \left(l_{2} \sin{\left(\theta_{2} \right)} + l_{3} \sin{\left(\theta_{2} + \theta_{3} \right)}\right) \cos{\left(\theta_{1} \right)} & - l_{3} \sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\\\left(l_{1} + l_{2} \cos{\left(\theta_{2} \right)} + l_{3} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \cos{\left(\theta_{1} \right)} & - \left(l_{2} \sin{\left(\theta_{2} \right)} + l_{3} \sin{\left(\theta_{2} + \theta_{3} \right)}\right) \sin{\left(\theta_{1} \right)} & - l_{3} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{2} + \theta_{3} \right)}\\0 & l_{2} \cos{\left(\theta_{2} \right)} + l_{3} \cos{\left(\theta_{2} + \theta_{3} \right)} & l_{3} \cos{\left(\theta_{2} + \theta_{3} \right)}\\0 & \sin{\left(\theta_{1} \right)} & \sin{\left(\theta_{1} \right)}\\0 & - \cos{\left(\theta_{1} \right)} & - \cos{\left(\theta_{1} \right)}\\1 & 0 & 0\end{matrix}\right]$

```python
det_Jv = sp.simplify(J[:3,:].det())
det_Jv
```

$\displaystyle - l_{2} l_{3} \left(l_{1} + l_{2} \cos{\left(\theta_{2} \right)} + l_{3} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \sin{\left(\theta_{3} \right)}$

```python
interval = sp.Interval(-sp.pi, sp.pi, left_open=True)
solution = sp.solveset(det_Jv, theta3, interval)
solution
```

$\displaystyle \left\{0, \pi\right\} \cup \left\{\theta_{3} \mid \theta_{3} \in \mathbb{R} \wedge \cos{\left(\theta_{2} + \theta_{3} \right)} + \frac{l_{1} + l_{2} \cos{\left(\theta_{2} \right)}}{l_{3}} = 0 \right\}$


```python
from sympy import *
init_printing() # Set up fancy printing
```

```python
# Mencari turunan fungsi menggunakan library sympy
from sympy import symbols, simplify, sqrt, latex

sb, sy, sC, sS, sQ = symbols('b, y, C, s, Q')
sA = sb * sy
sP = sb + 2* sy
sR = sA/sP
sfy = sA*sC*sqrt(sR*sS) - sQ

# Mencari Turunan f(y)
print('Persamaan Q: \n\tf(y)=', sfy, '\n==\n In Latex:', latex(sfy), '\n')
sfy_d = sfy.diff(sy)
print('Turunan Persamaan Q terhadap y : \n\tf\'(y)=', sfy_d, '\n==\n In Latex:', latex(sfy_d), '\n')
print('Bentuk sederhana : \n\tf\'(y)=', simplify(sfy_d), '\n==\n In Latex:', latex(simplify(sfy_d)), '\n')

# Mendefinisikan fungsi dalam python
def f(y):
    return float(sfy.subs({sS: S, sC: C, sb: b, sy: y, sQ: Q}))

def fd(y):
    return float(sfy_d.subs({sS: S, sC: C, sb: b, sy: y}))
```

    Persamaan Q:
    	f(y)= C*b*y*sqrt(b*s*y/(b + 2*y)) - Q
    ==
     In Latex: C b y \sqrt{\frac{b s y}{b + 2 y}} - Q

    Turunan Persamaan Q terhadap y :
    	f'(y)= C*b*sqrt(b*s*y/(b + 2*y)) + C*sqrt(b*s*y/(b + 2*y))*(b + 2*y)*(-b*s*y/(b + 2*y)**2 + b*s/(2*(b + 2*y)))/s
    ==
     In Latex: C b \sqrt{\frac{b s y}{b + 2 y}} + \frac{C}{s} \sqrt{\frac{b s y}{b + 2 y}} \left(b + 2 y\right) \left(- \frac{b s y}{\left(b + 2 y\right)^{2}} + \frac{b s}{2 \left(b + 2 y\right)}\right)

    Bentuk sederhana :
    	f'(y)= C*b*sqrt(b*s*y/(b + 2*y))*(3*b + 4*y)/(2*(b + 2*y))
    ==
     In Latex: \frac{C b \sqrt{\frac{b s y}{b + 2 y}}}{2 \left(b + 2 y\right)} \left(3 b + 4 y\right)

```python
print(latex(sfy.diff(sy).diff(sy).subs({sy: 5})))
```

    \frac{\sqrt{5} C}{s} \sqrt{\frac{b s}{b + 10}} \left(b + 10\right) \left(- \frac{2 b s}{\left(b + 10\right)^{2}} + \frac{20 b s}{\left(b + 10\right)^{3}}\right) + \frac{\sqrt{5} C}{5 s} \sqrt{\frac{b s}{b + 10}} \left(b + 10\right) \left(\frac{b s}{2 \left(b + 10\right)} - \frac{5 b s}{\left(b + 10\right)^{2}}\right) + \frac{2 C}{s} \sqrt{5} \sqrt{\frac{b s}{b + 10}} \left(\frac{b s}{2 \left(b + 10\right)} - \frac{5 b s}{\left(b + 10\right)^{2}}\right) + \frac{\sqrt{5} C \sqrt{\frac{b s}{b + 10}}}{5 b s^{2}} \left(b + 10\right)^{2} \left(\frac{b s}{2 \left(b + 10\right)} - \frac{5 b s}{\left(b + 10\right)^{2}}\right)^{2}

$$
Persamaan: C b \sqrt{\frac{b s y}{b + 2 y}} + \frac{C}{s} \sqrt{\frac{b s y}{b + 2 y}} \left(b + 2 y\right) \left(- \frac{b s y}{\left(b + 2 y\right)^{2}} + \frac{b s}{2 \left(b + 2 y\right)}\right)
$$

$$\begin{aligned}y &= 30\ m \\
y &= 10000\ m\end{aligned}
$$

$$\frac{\sqrt{5} C}{s} \sqrt{\frac{b s}{b + 10}} \left(b + 10\right) \left(- \frac{2 b s}{\left(b + 10\right)^{2}} + \frac{20 b s}{\left(b + 10\right)^{3}}\right) + \frac{\sqrt{5} C}{5 s} \sqrt{\frac{b s}{b + 10}} \left(b + 10\right) \left(\frac{b s}{2 \left(b + 10\right)} - \frac{5 b s}{\left(b + 10\right)^{2}}\right) + \frac{2 C}{s} \sqrt{5} \sqrt{\frac{b s}{b + 10}} \left(\frac{b s}{2 \left(b + 10\right)} - \frac{5 b s}{\left(b + 10\right)^{2}}\right) + \frac{\sqrt{5} C \sqrt{\frac{b s}{b + 10}}}{5 b s^{2}} \left(b + 10\right)^{2} \left(\frac{b s}{2 \left(b + 10\right)} - \frac{5 b s}{\left(b + 10\right)^{2}}\right)^{2}
$$

```python

```


# Lorenz System

The Lorenz system is a series of ordinary differential equations studied by MIT Professor Edward Lorenz:

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

It is an idealized model of atmospheric convection.  \\( x \\) is proportional to the rate of convection, \\( y \\) to the horizontal temperature variation, and \\( z \\) to the vertical temperature variation. \\( \sigma \\), \\( \rho \\), \\( \beta \\) are fluid dynamic parameters. Let's plot the Lorenz attractor using these equations.

Define our imports

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

Define our `lorenz` function to model equations defined earlier.

```python
def lorenz (x, y, z, s=10, r=28, b=2.667):
    x_dot = s*(y - x)
    y_dot = r*x - y - x*z
    z_dot = x*y - b*z
    return x_dot, y_dot, z_dot
```

Finally, let's use this function to plot the Lorenz Attractor:

```python
dt = 0.01; stepCnt = 10000
xs = np.empty((stepCnt + 1,))
ys = np.empty((stepCnt + 1,))
zs = np.empty((stepCnt + 1,))
xs[0], ys[0], zs[0] = (0., 1., 1.05)

for i in range(stepCnt):
    x_dot, y_dot, z_dot = lorenz(xs[i], ys[i], zs[i])
    xs[i + 1] = xs[i] + (x_dot * dt)
    ys[i + 1] = ys[i] + (y_dot * dt)
    zs[i + 1] = zs[i] + (z_dot * dt)

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot(xs, ys, zs, lw=0.5)
plt.title('Lorenz Attractor')
plt.show()
```

```python

```


# Classification Problem  (Assignment 5, TAO, Spring 2019)
### Instructor: Dr. Pawan Kumar

## Classification Problem
### Given a set of input vectors corresponding to objects (or featues) decide which of the N classes the object belogs to.

### Reference (some figures for illustration below are taken from this):
1. SVM without Tears, https://med.nyu.edu/chibi/sites/default/files/chibi/Final.pdf

## Possible Methods for Classification Problem
1. Perceptron
2. SVM
3. Neural Networks

## SVM: Support Vector Machines

We will briefly describe the idea behind support vector machines for classification problems. We first describe linear SVM used to classify linearly separable data, and then we describe how we can use these algorithm for non-linearly separable data by so called kernels. The kernels are functions that map non-linearly separable data to a space usually higher dimensional space where the data becomes linearly separable. Let us quickly start with linear SVM.

### Linear SVM for two class classification
We recall the separating hyperpplane theorem: If there are two non-intersecting convex set, then there exists a hyperplane that separates the two convex sets. This is the assumption we will make: we assume that the convex hull of the given data leads to two convex sets for two classes, such that a hyperplane exists that separates the convex hulls.

### Main idea of SVM:
Not just find a hyperplane (as in perceptrons), but find one that keeps a good (largest possible) gap from the the data samples of each class. This gap is popularly called margins.

### Illustration of problem, and kewords
Consider the dataset of cancer and normal patients, hence it is a two class problem. Let us visualize the data:

Let us notice the following about the given data:
0. There are two classes: blue shaded stars and red shaded circles.
2. The input vector is two dimensional, hence it is of the form $(x_1^1, x_2^1).$
2. Here $x_1^1, x_2^2$ are values of the features corresponding to two gene features: Gene X, Gene Y.
3. Here red line is the linear classifier or hyperplane that separates the given input data.
4. There are two dotted lines: one passes through a blue star point, and another dotted line passes through two red shaded circle points.
5. The distance between the two dotted lines is called gap or margin that we mentioned before.
6. Goal of SVM compared to perceptrons is to maximize this margin.

## Formulation of Optimization Model for Linear SVM
We now assume that the red hyperplane above with maximum margin is given by $$w \cdot x + b,$$
We further assume that the dotted lines above are given by $$w \cdot x + b = -1, \quad w \cdot x + b = +1.$$

For reasons, on why we can choose such hyperplane is shown in slides Lecture 16 of TAO. Since we want to maximize the margin the distance between the dotted lines, we recall the formula for diatance between planes. Let $D$ denote the distance, then
$$D = 2/ \| w \|.$$
So, to maximize the margin $D,$ we need to minimize $\| w \|.$ For convenience of writing algorithm (for differentiable function), we can say that minimizing $\| w \|$ is equivalent to minimizing $1/2 \| w \|^2.$ Hence
### Objective function: $\dfrac{1}{2} \| w \|^2$
For our hyperplane to classify correctly, we need points of one class on one side of dotted line, more concretely
$$w \cdot x + b \leq -1,$$
and the we want the samples of another class (red ones) be on the other side of other dotted lines, i.e.,
$$ w \cdot x + b \geq +1.$$
Let us now look what constraints mean in figure:

With this we are all set to write the constraints for our optimization model for SVM.
### Constraints:
$$
\begin{align}
&w \cdot x_i + b \leq -1, \quad \text{if}~y_i = -1\\
&w \cdot x_i + b \geq +1, \quad \text{if}~y_i = +1
\end{align}
$$
Hence, objective function with constraints, gives us the full model. The data for which the label $y_i$ is $-1$ satisfies $w \cdot x + b \leq -1,$ and the data for which the lable $y_i$ is $+1$ satisfies $w \cdot x + b \geq +1.$ Hence both these conditions can be combined to get
$$
\begin{align}
y_i (w \cdot x_i + b) \geq 1
\end{align}
$$

## Optimization Model (Primal Form):
$$
\begin{align}
\text{minimize} \quad & \dfrac{1}{2} \| w \|^2 \\
\text{subject to} \quad &w \cdot x_i + b \geq 1, \quad i=1,\dots,m,
\end{align}
$$
where $m$ is the number of samples $x_i,$ and $w \in \mathbb{R}^n.$

$\color{red}{Question:}$ Prove that the primal objective is convex.

$\color{blue}{Answer}:$ Put your answer here.

$\color{red}{Question}:$ Write the primal problem in standard form.

$\color{blue}{Answer}:$ Put your answer here.

## Optimization Model (Dual Form)
The dual form was derived in lecture 16:
$$
\begin{align*}
&\text{maximize} \quad \sum_{i=1}^m{\lambda_i} - \dfrac{1}{2} \sum_{i=1}^m \lambda_i \lambda_j y_i y_j (x_i \cdot x_j) \\
&\text{subject to} \quad \lambda_i \geq 0, \quad \sum_{i=1}^m{\lambda_i y_i} = 0, \quad i = 1, \dots, m
\end{align*},
$$
where $\lambda_i$ is the Lagrange multiplier. We claim that strong duality holds.

$\color{red}{Question:}$ Show the derivation of dual.

$\color{blue}{Answer:}$ Put your anser here (use latex)

$\color{red}{Question:}$ Prove that strong duality holds.

$\color{blue}{Answer:}$ Put your answer here (use latex)

$\color{red}{Question:}$ Prove that the dual objective is concave.

$\color{blue}{Answer}:$ Put your answer here.

$\color{red}{Question}:$ Write the dual problem in standard form.

$\color{blue}{Answer}:$ Put your answer here.

## Soft Margin SVM
In a variant of soft margin SVM, we assume that some data samples may be outliers or noise, and this prevents the data from being linearly separable. For example, see the figure below

In the figure, we see that

- We believe that two red and one blue sample is noisy or outliers.
- We now want to take into account that real life data is noisy, we decide to allow for some of the noisy data in the margin.
- Let $\xi_i$ denotes how far a data sample is from the middle plane (margin is the area between dotted line).
- For example, one of the noisy red data point in 0.6 away from middle red plane.
- We introduce this slack variable $\xi_i \geq 0$ for each data sample $x_i.$

## Optimization Model: Primal Soft-Margin
We can then write the primal soft-margin optimization model as follows:
$$
\begin{align*}
&\text{minimize} \quad \dfrac{1}{2} \| w \|^2 + C \sum_{i=1}^m \xi_i \\
&\text{subject to} \quad y_i (w \cdot x_i + b) \geq 1 - \xi_i, \quad \xi_i \geq 0, \quad i = 1, \dots, m.
\end{align*}
$$

## Optimization Model: Dual Soft-Margin
We can also write the dual form of soft-margin SVM as follows:
$$
\begin{align*}
\text{Maximize} \quad &\sum_{i=1}^m \lambda_i - \dfrac{1}{2} \sum_{i,j=1}^m \lambda_i \lambda_j \: y_i y_j \: x_i \cdot x_j \\
\text{subject to} \quad &0 \leq \lambda_i \leq C, \quad  i=1, \dots, m, \\
&\sum_{i=1}^m \lambda_i y_i = 0.
\end{align*}
$$

$\color{red}{Question:}$ Show the derivation of dual.

$\color{blue}{Answer:}$ Put your anser here (use latex)

$\color{red}{Question:}$ List advantages of dual over primal.

$\color{blue}{Answer:}$ Put your answer here (use latex)

# Kernels in SVM
## Non-Linear Classifiers
- For nonlinear data, we may map the data to a higher dimensional feature space where it is separable. See the figure below:

Such non-linear transformation can be implemented more effectively using the dual formulation.
- If we solve the dual form of linear SVM, then the predictions is given by
$$
\begin{align*}
f(x) &= \text{sign}(w \cdot x + b) \\
w &= \sum_{i=1}^m \alpha_i y_i x_i
\end{align*}
$$
If we assume that we did some transform $\Phi,$ then the classifier is given by
$$
\begin{align*}
f(x) &= \text{sign} (w \cdot \Phi(x) + b) \\
w &= \sum_{i=1}^m \alpha_i y_i \Phi(x_i)
\end{align*}
$$
If we substitute $w$ in $f(x),$ we observe that
$$
\begin{align*}
f(x) = \text{sign} \left ( \sum_{i=1}^m \alpha_i y_i \, \Phi(x_i) \cdot \Phi(x) + b \right) = \text{sign} \left( \sum_{i=1}^m \alpha_i y_i \, K(x_i, x) + b \right)
\end{align*}
$$
Note that doing dot products such as $\Phi(x_i) \cdot \Phi(x),$ if $\Phi(x)$ is a long vector! An important observation is to define this dot product or $K(x,z)$ such that dot products happen in input space rather than the feature space. We can see this with following example:
$$
\begin{align*}
K(x \cdot z) &= (x \cdot z)^2 = \left( \begin{bmatrix}
x_{(1)} \\ x_{(2)}
\end{bmatrix} \cdot \begin{bmatrix}
z_{(1)} \\ z_{(2)}
\end{bmatrix} \right)^2 = (x_{(1)} z_{(1)} + x_{(2)} z_{(2)})^2 \\
&= x_{(1)}^2 z_{(1)}^2 + 2x_{(1)} z_{(1)} x_{(2)} z_{(2)} + x_{(2)}^2 z_{(2)}^2 = \begin{bmatrix}
x_{(1)}^2 \\ \sqrt{2} x_{(1)} x_{(2)} \\ x_{(2)}^2
\end{bmatrix} \cdot \begin{bmatrix}
z_{(1)}^2 \\ \sqrt{2} z_{(1)} z_{(2)} \\ z_{(2)}^2
\end{bmatrix}  \\
&= \Phi(x) \cdot \Phi(z)
\end{align*}
$$

$\color{red}{Question:}$ Let the kernel be defined by $K(x,z) = (x \cdot z)^3.$ Define $\Phi(x).$ Assuming that one multiplications is 1 FLOP, and one addition is 1 FLOP, then how many flops you need to compute $K(x \cdot z)$ in input space versus feature space?

$\color{blue}{Answer:}$ Write your answer in this cell.

## Optimization Model: Dual Soft Margin Kernel SVM
We can now write the dual form of soft-margin Kernel SVM as follows:
$$
\begin{align*}
\text{Maximize} \quad &\sum_{i=1}^m \lambda_i - \dfrac{1}{2} \sum_{i, \, j=1}^m \lambda_i \lambda_j \: y_i y_j \: \Phi(x_i) \cdot \Phi(x_j) \\
\text{subject to} \quad &0 \leq \lambda_i \leq C, \quad  i=1, \dots, m, \\
&\sum_{i=1}^m \lambda_i y_i = 0.
\end{align*}
$$

## Solver for Optimization Problem: Quadratic Programming
We aspire to solve the above optimization problem using existing quadraric programming library. But we have a problem: the standard libraries use the standard form of the quadratic optimization problem that looks like the following:
$$
\begin{align*}
\text{minimize} \quad &\dfrac{1}{2} x^T P x + q^T x, \\
\text{subject to} \quad &Gx \leq h, \\
&Ax = b
\end{align*}
$$

# Dual Soft-Margin Kernel SVM in Standard QP: Assemble Matrices Vectors
To put the dual Kernel SVM in standard form, we need to set
- matrix $P$
- vector $x$
- vector $q$
- vector $h$
- vector $b$
- matrix $G$
- matrix $A$

### Matrix $P$
Let $$K(x_i, x_j) = \Phi(x_i) \cdot \Phi(x_j),$$ and set $(i,j)$ entry of matrix $P$ as $$P_{ij} = y_iy_j K(x_i,x_j)$$

## Vector $x$
Set $$x = \begin{bmatrix}
\lambda_1 \\
\lambda_2 \\
\vdots \\
\lambda_m
\end{bmatrix}
$$

## Vector $q$
Set $q \in \mathbb{R}^m$
$$ q =
\begin{bmatrix}
-1 \\ -1 \\ \vdots \\ -1
\end{bmatrix}
$$

## Matrix $A$
Set the matrix (in fact vector) $A$ as
$$
A = [y_1, y_2, \dots, y_m]
$$

## Vector $b$
In fact vector $b$ is a scalar here: $$b = 0$$

## Matrix $G$
$$
\begin{align*}
G = \begin{bmatrix}
1 & 0 & \dots & 0 \\
0 & 1 & \dots & 0 \\
\vdots & \ddots & \dots & \vdots \\
0 & 0 & \dots & 1 \\ \hline
-1 & 0 & \dots & 0 \\
\vdots & \ddots & \dots & \vdots \\
0 & 0 & \dots& -1
\end{bmatrix}
\end{align*}
$$

## Vector $h$
Set $h$ as
$$
h = \begin{bmatrix}
C \\
C \\
\vdots \\
C \\
0 \\
\vdots \\
0
\end{bmatrix}
$$

# Implementation of Kernel SVM
We are all set to try out coding the classifier using Kernel SVM. We will first import some libraries. Some of these libraries may not be available in your system. You may install them as follows:
- conda install numpy
- conda install -c conda-forge cvxopt
- sudo apt-get install python-scipy python-matplotlib

Try google search, if these does not work.

```python
import pylab as pl
import cvxopt as cvxopt
from cvxopt import solvers
import numpy as np
```

We will now define a class: svm
This class will have the following functions:
- __init__: where we will define initial default parameters
- *construct_kernel*: here we will define some kernels such as polynomial and RBF (radial basis or Gaussian kernel)
- *train_kernel_svm*: Here we will train, i.e, we will call a quadratic programming solver from cvxopt
- *classify*: Here we will test our classifier

$\color{red}{Question:}$ Fill the TODO below.

```python
class svm:

    def __init__(self, kernel='linear', C=None, sigma=1., degree=1., threshold=1e-5):
        self.kernel = kernel
        if self.kernel == 'linear':
            self.degree = 1.
            self.kernel = 'poly'

        self.C = C
        self.sigma = sigma
        self.threshold = threshold
        self.degree = degree

    def construct_kernel(self, X):
        self.K = np.dot(X, X.T)

        if self.kernel == 'poly':
            self.K = (1. + 1./self.sigma*self.K)**self.degree

        elif self.kernel == 'rbf':
            self.xsquared = (np.diag(self.K)*np.ones((1, self.N))).T
            b = np.ones((self.N, 1))
            self.K -= 0.5*(np.dot(self.xsquared, b.T) +
                           np.dot(b, self.xsquared.T))
            self.K = np.exp(self.K/(2.*self.sigma**2))

    def train_kernel_svm(self, X, targets):
        self.N = np.shape(X)[0]
        self.construct_kernel(X)

        # Assemble the matrices for the constraints
        P = TODO
        q = TODO
        G = TODO
        h = TODO
        A = TODO
        b = TODO

        # Call the the quadratic solver of cvxopt library.
        sol = cvxopt.solvers.qp(cvxopt.matrix(P), cvxopt.matrix(q), cvxopt.matrix(
            G), cvxopt.matrix(h), cvxopt.matrix(A), cvxopt.matrix(b))

        # Get the Lagrange multipliers out of the solution dictionary
        lambdas = np.array(sol['x'])

        # Find the (indices of the) support vectors, which are the vectors with non-zero Lagrange multipliers
        self.sv = np.where(lambdas > self.threshold)[0]
        self.nsupport = len(self.sv)
        print ("Number of support vectors = ", self.nsupport)

        # Keep the data corresponding to the support vectors
        self.X = X[self.sv, :]
        self.lambdas = lambdas[self.sv]
        self.targets = targets[self.sv]

        self.b = np.sum(self.targets)
        for n in range(self.nsupport):
            self.b -= np.sum(self.lambdas*self.targets *
                             np.reshape(self.K[self.sv[n], self.sv], (self.nsupport, 1)))
        self.b /= len(self.lambdas)

        if self.kernel == 'poly':
            def classify(Y, soft=False):
                K = (1. + 1./self.sigma*np.dot(Y, self.X.T))**self.degree

                self.y = np.zeros((np.shape(Y)[0], 1))
                for j in range(np.shape(Y)[0]):
                    for i in range(self.nsupport):
                        self.y[j] += self.lambdas[i]*self.targets[i]*K[j, i]
                    self.y[j] += self.b

                if soft:
                    return self.y
                else:
                    return np.sign(self.y)

        elif self.kernel == 'rbf':
            def classify(Y, soft=False):
                K = np.dot(Y, self.X.T)
                c = (1./self.sigma * np.sum(Y**2, axis=1)
                     * np.ones((1, np.shape(Y)[0]))).T
                c = np.dot(c, np.ones((1, np.shape(K)[1])))
                aa = np.dot(self.xsquared[self.sv],
                            np.ones((1, np.shape(K)[0]))).T
                K = K - 0.5*c - 0.5*aa
                K = np.exp(K/(2.*self.sigma**2))

                self.y = np.zeros((np.shape(Y)[0], 1))
                for j in range(np.shape(Y)[0]):
                    for i in range(self.nsupport):
                        self.y[j] += self.lambdas[i]*self.targets[i]*K[j, i]
                    self.y[j] += self.b

                if soft:
                    return self.y
                else:
                    return np.sign(self.y)
        else:
            print ("Error: Invalid kernel")
            return

        self.classify = classify

```

$\color{red}{Question:}$ How $b$ was computed?

$\color{blue}{Answer:}$ Write your answer here.

# Test the Classifier
In the following, we will now test our classifier.

```python
from importlib import reload
import pylab as pl
import numpy as np

iris = np.loadtxt('iris_proc.data', delimiter=',')
imax = np.concatenate((iris.max(axis=0)*np.ones((1, 5)),
                       iris.min(axis=0)*np.ones((1, 5))), axis=0).max(axis=0)

target = -np.ones((np.shape(iris)[0], 3), dtype=float)
indices = np.where(iris[:, 4] == 0)
target[indices, 0] = 1.
indices = np.where(iris[:, 4] == 1)
target[indices, 1] = 1.
indices = np.where(iris[:, 4] == 2)
target[indices, 2] = 1.

train = iris[::2, 0:4]
traint = target[::2]
test = iris[1::2, 0:4]
testt = target[1::2]

```

```python
# Training the machines
output = np.zeros((np.shape(test)[0], 3))

# Train for the first set of train data
#svm0 = svm(kernel='linear')
#svm0 = svm(kernel='linear')
#svm0 = svm.svm(kernel='poly',C=0.1,degree=1)
svm0 = svm(kernel='rbf')
svm0.train_kernel_svm(train, np.reshape(traint[:, 0], (np.shape(train[:, :2])[0], 1)))
output[:, 0] = svm0.classify(test, soft=True).T

# Train for the second set of train data
#svm1 = svm(kernel='linear')
#svm1 = svm(kernel='linear')
#svm1 = svm(kernel='poly',degree=3)
svm1 = svm(kernel='rbf')
svm1.train_kernel_svm(train, np.reshape(traint[:, 1], (np.shape(train[:, :2])[0], 1)))
output[:, 1] = svm1.classify(test, soft=True).T

# Train for the third set of train data
#svm2 = svm(kernel='linear')
#svm2 = svm(kernel='linear')
#svm2 = svm(kernel='poly',C=0.1,degree=1)
svm2 = svm(kernel='rbf')
svm2.train_kernel_svm(train, np.reshape(traint[:, 2], (np.shape(train[:, :2])[0], 1)))
output[:, 2] = svm2.classify(test, soft=True).T
```

```python
# Make a decision about which class
# Pick the one with the largest margin
bestclass = np.argmax(output, axis=1)
print (bestclass)
print (iris[1::2, 4])
print("Misclassified locations:")
err = np.where(bestclass != iris[1::2, 4])[0]
print (err)
print (float(np.shape(testt)[0] - len(err)) /
       (np.shape(testt)[0]), "test accuracy")
```

## Further Questions

$\color{red}{Question}:$ The IRIS dataset has three classes. Explain by observing the code above how the two class SVM was modified for multiclass classification.

$\color{red}{Question}:$ Write mathematical expressions for the kernels defined above.

$\color{red}{Question}:$ Play with different Kernels. Which kernels (polynomial, RBF, or polynomial) give the best test accuracy?