# Chapter 5

`Original content created by Cam Davidson-Pilon`

`Ported to Python 3 and PyMC3 by Max Margenot (@clean_utensils) and Thomas Wiecki (@twiecki) at Quantopian (@quantopian)`
____

### Would you rather lose an arm or a leg?

Statisticians can be a sour bunch. Instead of considering their winnings, they only measure how much they have lost. In fact, they consider their wins as *negative losses*. But what's interesting is *how they measure their losses.*

For example, consider the following example:

>   A meteorologist is predicting the probability of a possible hurricane striking his city. He estimates, with 95% confidence, that the probability of it *not* striking is between 99% - 100%. He is very happy with his precision and advises the city that a major evacuation is unnecessary. Unfortunately, the hurricane does strike and the city is flooded.

This stylized example shows the flaw in using a pure accuracy metric to measure outcomes. Using a measure that emphasizes estimation accuracy, while an appealing and *objective* thing to do, misses the point of why you are even performing the statistical inference in the first place: results of inference. The author Nassim Taleb of *The Black Swan* and *Antifragility* stresses the importance of the *payoffs* of decisions, *not the accuracy*. Taleb distills this quite succinctly: "I would rather be vaguely right than very wrong."

## Loss Functions

We introduce what statisticians and decision theorists call *loss functions*. A loss function is a function of the true parameter, and an estimate of that parameter

$$ L( \theta, \hat{\theta} ) = f( \theta, \hat{\theta} )$$

The important point of loss functions is that it measures how *bad* our current estimate is: the larger the loss, the worse the estimate is according to the loss function. A simple, and very common, example of a loss function is the *squared-error loss*:

$$ L( \theta, \hat{\theta} ) = ( \theta -  \hat{\theta} )^2$$

The squared-error loss function is used in estimators like linear regression, UMVUEs and many areas of machine learning. We can also consider an asymmetric squared-error loss function, something like:

$$ L( \theta, \hat{\theta} ) = \begin{cases} ( \theta -  \hat{\theta} )^2 & \hat{\theta} \lt \theta \\\\ c( \theta -  \hat{\theta} )^2 & \hat{\theta} \ge \theta, \;\; 0\lt c \lt 1 \end{cases}$$

which represents that estimating a value larger than the true estimate is preferable to estimating a value below. A situation where this might be useful is in estimating web traffic for the next month, where an over-estimated outlook is preferred so as to avoid an underallocation of server resources.

A negative property about the squared-error loss is that it puts a disproportionate emphasis on large outliers. This is because the loss increases quadratically, and not linearly, as the estimate moves away. That is, the penalty of being three units away is much less than being five units away, but the penalty is not much greater than being one unit away, though in both cases the magnitude of difference is the same:

$$ \frac{1^2}{3^2} \lt \frac{3^2}{5^2}, \;\; \text{although} \;\; 3-1 = 5-3 $$

This loss function imposes that large errors are *very* bad. A more *robust* loss function that increases linearly with the difference is the *absolute-loss*

$$ L( \theta, \hat{\theta} ) = | \theta -  \hat{\theta} | $$

Other popular loss functions include:

-  $L( \theta, \hat{\theta} ) = \mathbb{1}_{ \hat{\theta} \neq \theta }$ is the zero-one loss often used in machine learning classification algorithms.
-  $L( \theta, \hat{\theta} ) = -\theta\log( \hat{\theta} ) - (1- \theta)\log( 1 - \hat{\theta} ), \; \; \theta \in {0,1}, \; \hat{\theta} \in [0,1]$, called the *log-loss*, also used in machine learning.

Historically, loss functions have been motivated from 1) mathematical convenience, and 2) they are robust to application, i.e., they are objective measures of loss. The first reason has really held back the full breadth of loss functions. With computers being agnostic to mathematical convenience, we are free to design our own loss functions, which we take full advantage of later in this Chapter.

With respect to the second point, the above loss functions are indeed objective, in that they are most often a function of the difference between estimate and true parameter, independent of signage or payoff of choosing that estimate. This last point, its independence of payoff, causes quite pathological results though. Consider our hurricane example above: the statistician equivalently predicted that the probability of the hurricane striking was between 0% to 1%. But if he had ignored being precise and instead focused on outcomes (99% chance of no flood, 1% chance of flood), he might have advised differently.

By shifting our focus from trying to be incredibly precise about parameter estimation to focusing on the outcomes of our parameter estimation, we can customize our estimates to be optimized for our application. This requires us to design new loss functions that reflect our goals and outcomes. Some examples of more interesting loss functions:

- $L( \theta, \hat{\theta} ) = \frac{ | \theta - \hat{\theta} | }{ \theta(1-\theta) }, \; \; \hat{\theta}, \theta \in [0,1]$ emphasizes an estimate closer to 0 or 1 since if the true value $\theta$ is near 0 or 1, the loss will be *very* large unless $\hat{\theta}$ is similarly close to 0 or 1.
This loss function might be used by a political pundit whose job requires him or her to give confident "Yes/No" answers. This loss reflects that if the true parameter is close to 1 (for example, if a political outcome is very likely to occur), he or she would want to strongly agree as to not look like a skeptic.

-  $L( \theta, \hat{\theta} ) =  1 - \exp \left( -(\theta -  \hat{\theta} )^2 \right)$ is bounded between 0 and 1 and reflects that the user is indifferent to sufficiently-far-away estimates. It is similar to the zero-one loss above, but not quite as penalizing to estimates that are close to the true parameter.
-  Complicated non-linear loss functions can programmed:

        def loss(true_value, estimate):
            if estimate*true_value > 0:
                return abs(estimate - true_value)
            else:
               return abs(estimate)*(estimate - true_value)**2

-  Another example is from the book *The Signal and The Noise*. Weather forecasters have an interesting loss function for their predictions.

>  People notice one type of mistake — the failure to predict rain — more than other, false alarms. If it rains when it isn't supposed to, they curse the weatherman for ruining their picnic, whereas an unexpectedly sunny day is taken as a serendipitous bonus.

>  [The Weather Channel's bias] is limited to slightly exaggerating the probability of rain when it is unlikely to occur — saying there is a 20 percent change when they know it is really a 5 or 10 percent chance — covering their butts in the case of an unexpected sprinkle.

As you can see, loss functions can be used for good and evil: with great power, comes great — well you know.

##  Loss functions in the real world

So far we have been under the unrealistic assumption that we know the true parameter. Of course if we knew the true parameter, bothering to guess an estimate is pointless. Hence a loss function is really only practical when the true parameter is unknown.

In Bayesian inference, we have a mindset that the unknown parameters are really random variables with prior and posterior distributions. Concerning the posterior distribution, a value drawn from it is a *possible* realization of what the true parameter could be. Given that realization, we can compute a loss associated with an estimate. As we have a whole distribution of what the unknown parameter could be (the posterior), we should be more interested in computing the *expected loss* given an estimate. This expected loss is a better estimate of the true loss than comparing the given loss from only a single sample from the posterior.

First it will be useful to explain a *Bayesian point estimate*. The systems and machinery present in the modern world are not built to accept posterior distributions as input. It is also rude to hand someone over a distribution when all they asked for was an estimate.  In the course of an individual's day, when faced with uncertainty we still act by distilling our uncertainty down to a single action. Similarly, we need to distill our posterior distribution down to a single value (or vector in the multivariate case). If the value is chosen intelligently, we can avoid the flaw of frequentist methodologies that mask the uncertainty and provide a more informative result.The value chosen, if from a Bayesian posterior, is a Bayesian point estimate.

Suppose $P(\theta | X)$ is the posterior distribution of $\theta$ after observing data $X$, then the following function is understandable as the *expected loss of choosing estimate $\hat{\theta}$ to estimate $\theta$*:

$$ l(\hat{\theta} ) = E_{\theta}\left[ \; L(\theta, \hat{\theta}) \; \right] $$

This is also known as the *risk* of estimate $\hat{\theta}$. The subscript $\theta$ under the expectation symbol is used to denote that $\theta$ is the unknown (random) variable in the expectation, something that at first can be difficult to consider.

We spent all of last chapter discussing how to approximate expected values. Given $N$ samples $\theta_i,\; i=1,...,N$ from the posterior distribution, and a loss function $L$, we can approximate the expected loss of using estimate $\hat{\theta}$ by the Law of Large Numbers:

$$\frac{1}{N} \sum_{i=1}^N \;L(\theta_i, \hat{\theta} ) \approx E_{\theta}\left[ \; L(\theta, \hat{\theta}) \; \right]  = l(\hat{\theta} ) $$

Notice that measuring your loss via an *expected value* uses more information from the distribution than the MAP estimate which, if you recall, will only find the maximum value of the distribution and ignore the shape of the distribution. Ignoring information can over-expose yourself to tail risks, like the unlikely hurricane, and leaves your estimate ignorant of how ignorant you really are about the parameter.

Similarly, compare this with frequentist methods, that traditionally only aim to minimize the error, and do not consider the *loss associated with the result of that error*. Compound this with the fact that frequentist methods are almost guaranteed to never be absolutely accurate. Bayesian point estimates fix this by planning ahead: your estimate is going to be wrong, you might as well err on the right side of wrong.

##### Example: Optimizing for the *Showcase* on *The Price is Right*

Bless you if you are ever chosen as a contestant on the Price is Right, for here we will show you how to optimize your final price on the *Showcase*. For those who forget the rules:

1. Two contestants compete in *The Showcase*.
2. Each contestant is shown a unique suite of prizes.
3. After the viewing, the contestants are asked to bid on the price for their unique suite of prizes.
4. If a bid price is over the actual price, the bid's owner is disqualified from winning.
5. If a bid price is under the true price by less than $250, the winner is awarded both prizes.

The difficulty in the game is balancing your uncertainty in the prices, keeping your bid low enough so as to not bid over, and trying to bid close to the price.

Suppose we have recorded the *Showcases* from previous *The Price is Right* episodes and have *prior* beliefs about what distribution the true price follows. For simplicity, suppose it follows a Normal:

$$\text{True Price} \sim \text{Normal}(\mu_p, \sigma_p )$$

In a later chapter, we will actually use *real Price is Right Showcase data* to form the historical prior, but this requires some advanced PyMC3 use so we will not use it here. For now, we will assume $\mu_p = 35 000$ and $\sigma_p = 7500$.

We need a model of how we should be playing the *Showcase*. For each prize in the prize suite, we have an idea of what it might cost, but this guess could differ significantly from the true price. (Couple this with increased pressure being onstage and you can see why some bids are so wildly off). Let's suppose your beliefs about the prices of prizes also follow Normal distributions:

$$ \text{Prize}_i \sim \text{Normal}(\mu_i, \sigma_i ),\;\; i=1,2$$

This is really why Bayesian analysis is great: we can specify what we think a fair price is through the $\mu_i$ parameter, and express uncertainty of our guess in the $\sigma_i$ parameter.

We'll assume two prizes per suite for brevity, but this can be extended to any number.
The true price of the prize suite is then given by $\text{Prize}_1 + \text{Prize}_2 + \epsilon$,
where $\epsilon$ is some error term.

We are interested in the updated $\text{True Price}$ given we have observed both prizes and have belief distributions about them. We can perform this using PyMC3.

Lets make some values concrete. Suppose there are two prizes in the observed prize suite:

1. A trip to wonderful Toronto, Canada!
2. A lovely new snowblower!

We have some guesses about the true prices of these objects, but we are also pretty uncertain about them. I can express this uncertainty through the parameters of the Normals:

\begin{align}
& \text{snowblower} \sim \text{Normal}(3 000, 500 )\\\\
& \text{Toronto} \sim \text{Normal}(12 000, 3000 )\\\\
\end{align}

For example, I believe that the true price of the trip to Toronto is 12 000 dollars, and that there is a 68.2% chance the price falls 1 standard deviation away from this, i.e. my confidence is that there is a 68.2% chance the trip is in [9 000, 15 000].

We can create some PyMC3 code to perform inference on the true price of the suite.

```python
%matplotlib inline
import scipy.stats as stats
from IPython.core.pylabtools import figsize
import numpy as np
import matplotlib.pyplot as plt

figsize(12.5, 9)

norm_pdf = stats.norm.pdf

plt.subplot(311)
x = np.linspace(0, 60000, 200)
# Historical total prices
sp1 = plt.fill_between(x , 0, norm_pdf(x, 35000, 7500),
                color = "#348ABD", lw = 3, alpha = 0.6,
                label = "historical total prices")
p1 = plt.Rectangle((0, 0), 1, 1, fc=sp1.get_facecolor()[0])
plt.legend([p1], [sp1.get_label()])

plt.subplot(312)
x = np.linspace(0, 10000, 200)
# Prior for the snowblower
sp2 = plt.fill_between(x , 0, norm_pdf(x, 3000, 500),
                 color = "#A60628", lw = 3, alpha = 0.6,
                 label="snowblower price guess")

p2 = plt.Rectangle((0, 0), 1, 1, fc=sp2.get_facecolor()[0])
plt.legend([p2], [sp2.get_label()])

plt.subplot(313)
x = np.linspace(0, 25000, 200)
# Prior for the trip
sp3 = plt.fill_between(x , 0, norm_pdf(x, 12000, 3000),
                 color = "#7A68A6", lw = 3, alpha = 0.6,
                 label = "Trip price guess")
plt.autoscale(tight=True)
p3 = plt.Rectangle((0, 0), 1, 1, fc=sp3.get_facecolor()[0])
plt.legend([p3], [sp3.get_label()]);
```

```python
import pymc3 as pm

# Prior params for prize prices (normal distribution)
data_mu = [3e3, 12e3]
data_std =  [5e2, 3e3]

# Historical params for the total showcase (normal distribution)
mu_prior = 35e3
std_prior =  75e2

with pm.Model() as model:
    # Historical total price distribution
    true_price = pm.Normal("true_price", mu=mu_prior, sd=std_prior)

    # Priors for individual prize prices
    prize_1 = pm.Normal("first_prize", mu=data_mu[0], sd=data_std[0])
    prize_2 = pm.Normal("second_prize", mu=data_mu[1], sd=data_std[1])
    price_estimate = prize_1 + prize_2  # Deterministic

    # Deterministic - get log prob density from our prior @ the sampled historical price
    logp = pm.Normal.dist(mu=price_estimate, sd=(3e3)).logp(true_price)
    error = pm.Potential("error", logp) # TODO: Understand this

    trace = pm.sample(50000, step=pm.Metropolis())
    burned_trace = trace[10000:]

price_trace = burned_trace["true_price"]
```

```python
figsize(12.5, 4)

import scipy.stats as stats

x = np.linspace(5000, 40000)
# Historical total price
plt.plot(x, stats.norm.pdf(x, 35000, 7500), c = "k", lw = 2,
         label = "prior dist. of suite price")

_hist = plt.hist(price_trace, bins = 35, normed= True, histtype= "stepfilled")
plt.title("Posterior of the true price estimate")
plt.vlines(mu_prior, 0, 1.1*np.max(_hist[0]), label = "prior's mean",
           linestyles="--")
plt.vlines(price_trace.mean(), 0, 1.1*np.max(_hist[0]), \
           label = "posterior's mean", linestyles="-.")
plt.legend(loc = "upper left");
```

Notice that because of our two observed prizes and subsequent guesses (including uncertainty about those guesses), we shifted our mean price estimate down about \$15,000 dollars from the previous mean price.

A frequentist, seeing the two prizes and having the same beliefs about their prices, would bid $\mu_1 + \mu_2 = 35000$, regardless of any uncertainty. Meanwhile, the *naive Bayesian* would simply pick the mean of the posterior distribution. But we have more information about our eventual outcomes; we should incorporate this into our bid. We will use the loss function above to find the *best* bid (*best* according to our loss).

What might a contestant's loss function look like? I would think it would look something like:

    def showcase_loss(guess, true_price, risk = 80000):
        if true_price < guess:
            return risk
        elif abs(true_price - guess) <= 250:
            return -2*np.abs(true_price)
        else:
            return np.abs(true_price - guess - 250)

where `risk` is a parameter that defines of how bad it is if your guess is over the true price. A lower `risk` means that you are more comfortable with the idea of going over. If we do bid under and the difference is less than $250, we receive both prizes (modeled here as receiving twice the original prize). Otherwise, when we bid under the `true_price` we want to be as close as possible, hence the `else` loss is a increasing function of the distance between the guess and true price.

For every possible bid, we calculate the *expected loss* associated with that bid. We vary the `risk` parameter to see how it affects our loss:

```python
figsize(12.5, 7)

"""
The loss from a guess knowing the true_price
"""
def showdown_loss(guess, true_price, risk = 80000):
        loss = np.zeros_like(true_price)
        ix = true_price < guess
        loss[~ix] = np.abs(guess - true_price[~ix])
        close_mask = [abs(true_price - guess) <= 250]
        loss[close_mask] = -2*true_price[close_mask]
        loss[ix] = risk
        return loss

guesses = np.linspace(5000, 50000, 70)
risks = np.linspace(30000, 150000, 6)

# The average loss for the guess. Single guess, average over the possible true prices
# price_trace is the posterior distribution of the total price of the prizes
expected_loss = lambda guess, risk: showdown_loss(guess, price_trace, risk).mean()

for _p in risks:
    results = [expected_loss(_g, _p) for _g in guesses]
    plt.plot(guesses, results, label = "%d"%_p)

plt.title("Expected loss of different guesses, \nvarious risk-levels of \
overestimating")
plt.legend(loc="upper left", title="Risk parameter")
plt.xlabel("price bid")
plt.ylabel("expected loss")
plt.xlim(5000, 30000);
```

### Minimizing our losses

It would be wise to choose the estimate that minimizes our expected loss. This corresponds to the minimum point on each of the curves above. More formally, we would like to minimize our expected loss by finding the solution to

$$ \text{arg} \min_{\hat{\theta}} \;\;E_{\theta}\left[ \; L(\theta, \hat{\theta}) \; \right] $$

The minimum of the expected loss is called the *Bayes action*. We can solve for the Bayes action using Scipy's optimization routines. The function in `fmin` in `scipy.optimize` module uses an intelligent search to find a minimum (not necessarily a *global* minimum) of any uni- or multivariate function. For most purposes, `fmin` will provide you with a good answer.

We'll compute the minimum loss for the *Showcase* example above:

```python
import scipy.optimize as sop

ax = plt.subplot(111)

for _p in risks:
    _color = next(ax._get_lines.prop_cycler)
    _min_results = sop.fmin(expected_loss, 15000, args=(_p,),disp = False)
    _results = [expected_loss(_g, _p) for _g in guesses]
    plt.plot(guesses, _results , color = _color['color'])
    plt.scatter(_min_results, 0, s = 60, \
                color= _color['color'], label = "%d"%_p)
    plt.vlines(_min_results, 0, 120000, color = _color['color'], linestyles="--")
    print("minimum at risk %d: %.2f" % (_p, _min_results))

plt.title("Expected loss & Bayes actions of different guesses, \n \
various risk-levels of overestimating")
plt.legend(loc="upper left", scatterpoints = 1, title = "Bayes action at risk:")
plt.xlabel("price guess")
plt.ylabel("expected loss")
plt.xlim(7000, 30000)
plt.ylim(-1000, 80000);
```

As intuition suggests, as we decrease the risk threshold (care about overbidding less), we increase our bid, willing to edge closer to the true price. It is interesting how far away our optimized loss is from the posterior mean, which was about 20 000.

Suffice to say, in higher dimensions being able to eyeball the minimum expected loss is impossible. Hence why we require use of Scipy's `fmin` function.

______

### Shortcuts

For some loss functions, the Bayes action is known in closed form. We list some of them below:

-  If using the mean-squared loss, the Bayes action is the mean of the posterior distribution, i.e. the value $$ E_{\theta}\left[ \theta \right] $$ minimizes $E_{\theta}\left[ \; (\theta - \hat{\theta})^2 \; \right]$. Computationally this requires us to calculate the average of the posterior samples [See chapter 4 on The Law of Large Numbers]

-  Whereas the *median* of the posterior distribution minimizes the expected absolute-loss. The sample median of the posterior samples is an appropriate and very accurate approximation to the true median.

-  In fact, it is possible to show that the MAP estimate is the solution to using a loss function that shrinks to the zero-one loss.

Maybe it is clear now why the first-introduced loss functions are used most often in the mathematics of Bayesian inference: no complicated optimizations are necessary. Luckily, we have machines to do the complications for us.

##  Machine Learning via Bayesian Methods

Whereas frequentist methods strive to achieve the best precision about all possible parameters, machine learning cares to achieve the best *prediction* among all possible parameters. Of course, one way to achieve accurate predictions is to aim for accurate predictions, but often your prediction measure and what frequentist methods are optimizing for are very different.

For example, least-squares linear regression is the most simple active machine learning algorithm. I say active as it engages in some learning, whereas predicting the sample mean is technically *simpler*, but is learning very little if anything. The loss that determines the coefficients of the regressors is a squared-error loss. On the other hand, if your prediction loss function (or score function, which is the negative loss) is not a squared-error, like AUC, ROC, precision, etc., your least-squares line will not be optimal for the prediction loss function. This can lead to prediction results that are suboptimal.

Finding Bayes actions is equivalent to finding parameters that optimize *not parameter accuracy* but an arbitrary performance measure, however we wish to define performance (loss functions, AUC, ROC, precision/recall etc.).

The next two examples demonstrate these ideas. The first example is a linear model where we can choose to predict using the least-squares loss or a novel, outcome-sensitive loss.

The second example is adapted from a Kaggle data science project. The loss function associated with our predictions is incredibly complicated.

##### Example: Financial prediction

Suppose the future return of a stock price is very small, say 0.01 (or 1%). We have a model that predicts the stock's future price, and our profit and loss is directly tied to us acting on the prediction.  How should we measure the loss associated with the model's predictions, and subsequent future predictions? A squared-error loss is agnostic to the signage and would penalize a prediction of -0.01 equally as bad a prediction of 0.03:

$$ (0.01 - (-0.01))^2 = (0.01 - 0.03)^2 = 0.004$$

If you had made a bet based on your model's prediction, you would have earned money with a prediction of 0.03, and lost money with a prediction of -0.01, yet our loss did not capture this. We need a better loss that takes into account the *sign* of the prediction and true value. We design a new loss that is better for financial applications below:

```python
figsize(12.5, 4)

def stock_loss(true_return, yhat, alpha = 100.):
    if true_return * yhat < 0:
        #opposite signs, not good
        return alpha*yhat**2 - np.sign(true_return)*yhat \
                        + abs(true_return)
    else:
        return abs(true_return - yhat)

true_value = .05
pred = np.linspace(-.04, .12, 75)

plt.plot(pred, [stock_loss(trhttps://www.youtube.com/watch?v=WG9MtK2nxPIhttps://www.youtube.com/watch?v=WG9MtK2nxPIue_value, _p) for _p in pred], \
        label = "Loss associated with\n prediction if true value = 0.05", lw =3)
plt.vlines(0, 0, .25, linestyles="--")

plt.xlabel("prediction")
plt.ylabel("loss")
plt.xlim(-0.04, .12)
plt.ylim(0, 0.25)

true_value = -.02
plt.plot(pred, [stock_loss(true_value, _p) for _p in pred], alpha = 0.6, \
        label = "Loss associated with\n prediction if true value = -0.02", lw =3)
plt.legend()
plt.title("Stock returns loss if true value = 0.05, -0.02");
```

Note the change in the shape of the loss as the prediction crosses zero. This loss reflects that the user really does not want to guess the wrong sign, especially be wrong *and* a large magnitude.

Why would the user care about the magnitude? Why is the loss not 0 for predicting the correct sign? Surely, if the return is 0.01 and we bet millions we will still be (very) happy.

Financial institutions treat downside risk, as in predicting a lot on the wrong side, and upside risk, as in predicting a lot on the right side, similarly. Both are seen as risky behaviour and discouraged. Hence why we have an increasing loss as we move further away from the true price. (With less extreme loss in the direction of the correct sign.)

We will perform a regression on a trading signal that we believe predicts future returns well. Our dataset is artificial, as most financial data is not even close to linear. Below, we plot the data along with the least-squares line.

```python
## Code to create artificial data
N = 100
X = 0.025*np.random.randn(N)
Y = 0.5*X + 0.01*np.random.randn(N)

ls_coef_ = np.cov(X, Y)[0,1]/np.var(X)  # TODO: Look at this derivation
ls_intercept = Y.mean() - ls_coef_*X.mean()  # TODO: Look at this derivation

plt.scatter(X, Y, c="k")
plt.xlabel("trading signal")
plt.ylabel("returns")
plt.title("Empirical returns vs trading signal")
plt.plot(X, ls_coef_*X + ls_intercept, label = "Least-squares line")
plt.xlim(X.min(), X.max())
plt.ylim(Y.min(), Y.max())
plt.legend(loc="upper left");
```

We perform a simple Bayesian linear regression on this dataset. We look for a model like:

$$ R = \alpha + \beta x + \epsilon$$

where $\alpha, \beta$ are our unknown parameters and $\epsilon \sim \text{Normal}(0, \sigma)$. The most common priors on $\beta$ and $\alpha$ are Normal priors. We will also assign a prior on $\sigma$, so that $\sigma$ is uniform over 0 to 100.

```python
import pymc3 as pm

with pm.Model() as model:
    std = pm.Uniform("std", 0, 100)

    beta = pm.Normal("beta", mu=0, sd=100)
    alpha = pm.Normal("alpha", mu=0, sd=100)

    mean = pm.Deterministic("mean", alpha + beta*X)

    obs = pm.Normal("obs", mu=mean, sd=std, observed=Y)

    trace = pm.sample(100000, step=pm.Metropolis())
    burned_trace = trace[20000:]
```

```python
pm.plots.traceplot(burned_trace, var_names=["std", "beta", "alpha"])
pm.plot_posterior(burned_trace, var_names=["std", "beta", "alpha"]);
```

It appears the MCMC has converged so we may continue.

For a specific trading signal, call it $x$, the distribution of possible returns has the form:

$$R_i(x) =  \alpha_i + \beta_ix + \epsilon $$

where $\epsilon \sim \text{Normal}(0, \sigma_i)$ and $i$ indexes our posterior samples. We wish to find the solution to

$$ \arg \min_{r} \;\;E_{R(x)}\left[ \; L(R(x), r) \; \right] $$

according to the loss given above. This $r$ (think this should be the full equation) is our Bayes action for trading signal $x$. Below we plot the Bayes action over different trading signals. What do you notice?

```python
figsize(12.5, 6)
from scipy.optimize import fmin

"""
Loss, depends on the prediction and actual prices
"""
def stock_loss(price, pred, coef = 500):
    """vectorized for numpy"""
    sol = np.zeros_like(price)
    ix = price*pred < 0
    sol[ix] = coef*pred**2 - np.sign(price[ix])*pred + abs(price[ix])  # If get wrong sign then increase loss size
    sol[~ix] = abs(price[~ix] - pred)
    return sol

# Samples from the posterior
std_samples = burned_trace["std"]  # Standard dev of the noise
alpha_samples = burned_trace["alpha"]  # Intercept
beta_samples = burned_trace["beta"]  # Gradient

N = std_samples.shape[0]  # Num samples from the posterior
noise = std_samples*np.random.randn(N)  # Draw samples from normal distribution with sigmas taken from posterior

# Deterministic equation to calc predictions from input signal
possible_outcomes = lambda signal: alpha_samples + beta_samples*signal + noise

opt_predictions = np.zeros(50)
trading_signals =  np.linspace(X.min(), X.max(), 50)  # Range of input signals
pred = np.linspace(-.04, .12, 75)  # Range of possible predictions
for i, _signal in enumerate(trading_signals):
        _possible_outcomes = possible_outcomes(_signal)  # Get predictions from posterior
        tomin = lambda pred: stock_loss(_possible_outcomes, pred).mean()  # Find mean loss between prediction & possible output
        opt_predictions[i] = fmin(tomin, 0, disp = False)

plt.xlabel("trading signal")
plt.ylabel("prediction")
plt.title("Least-squares prediction vs. Bayes action prediction")
plt.plot(X, ls_coef_*X + ls_intercept, label ="Least-squares prediction")
plt.xlim(X.min(), X.max())
plt.plot(trading_signals, opt_predictions, label ="Bayes action prediction")
plt.legend(loc="upper left");
```

What is interesting about the above graph is that when the signal is near 0, and many of the possible  returns outcomes are possibly both positive and negative, our best (with respect to our loss) prediction is to predict close to 0, hence *take on no position*. Only when we are very confident do we enter into a position. I call this style of model a *sparse prediction*, where we feel uncomfortable with our uncertainty so choose not to act. (Compare with the least-squares prediction which will rarely, if ever, predict zero).

A good sanity check that our model is still reasonable: as the signal becomes more and more extreme, and we feel more and more confident about the positive/negativeness of returns, our position converges with that of the least-squares line.

The sparse-prediction model is not trying to *fit* the data the best (according to a *squared-error loss* definition of *fit*). That honor would go to the least-squares model. The sparse-prediction model is trying to find the best prediction *with respect to our `stock_loss`-defined loss*. We can turn this reasoning around: the least-squares model is not trying to *predict* the best (according to a *`stock-loss`* definition of *predict*). That honor would go the *sparse prediction* model. The least-squares model is trying to find the best fit of the data *with respect to the squared-error loss*.

##### Example: Kaggle contest on *Observing Dark World*

A personal motivation for learning Bayesian methods was trying to piece together the winning solution to Kaggle's [*Observing Dark Worlds*](http://www.kaggle.com/c/DarkWorlds) contest. From the contest's website:

>There is more to the Universe than meets the eye. Out in the cosmos exists a form of matter that outnumbers the stuff we can see by almost 7 to 1, and we don’t know what it is. What we do know is that it does not emit or absorb light, so we call it Dark Matter. Such a vast amount of aggregated matter does not go unnoticed. In fact we observe that this stuff aggregates and forms massive structures called Dark Matter Halos. Although dark, it warps and bends spacetime such that any light from a background galaxy which passes close to the Dark Matter will have its path altered and changed. This bending causes the galaxy to appear as an ellipse in the sky.

The contest required predictions about where dark matter was likely to be. The winner, [Tim Salimans](http://timsalimans.com/), used Bayesian inference to find the best locations for the halos (interestingly, the second-place winner also used Bayesian inference). With Tim's permission, we provided his solution [1] here:

1. Construct a prior distribution for the halo positions $p(x)$, i.e. formulate our expectations about the halo positions before looking at the data.
2. Construct a probabilistic model for the data (observed ellipticities of the galaxies) given the positions of the dark matter halos: $p(e | x)$.
3. Use Bayes’ rule to get the posterior distribution of the halo positions, i.e. use to the data to guess where the dark matter halos might be.
4. Minimize the expected loss with respect to the posterior distribution over the predictions for the halo positions: $\hat{x} = \arg \min_{\text{prediction} } E_{p(x|e)}[ L( \text{prediction}, x) ]$ , i.e. tune our predictions to be as good as possible for the given error metric.

The loss function in this problem is very complicated. For the very determined, the loss function is contained in the file DarkWorldsMetric.py in the parent folder. Though I suggest not reading it all, suffice to say the loss function is about 160 lines of code — not something that can be written down in a single mathematical line. The loss function attempts to measure the accuracy of prediction, in a Euclidean distance sense, such that no shift-bias is present. More details can be found on the metric's [main page](http://www.kaggle.com/c/DarkWorlds/details/evaluation).

We will attempt to implement Tim's winning solution using PyMC3 and our knowledge of loss functions.

### The Data

The dataset is actually 300 separate files, each representing a sky. In each file, or sky, are between 300 and 720 galaxies. Each galaxy has an $x$ and $y$ position associated with it, ranging from 0 to 4200, and measures of ellipticity: $e_1$ and $e_2$. Information about what these measures mean can be found [here](https://www.kaggle.com/c/DarkWorlds/details/an-introduction-to-ellipticity), but for our purposes it does not matter besides for visualization purposes. Thus a typical sky might look like the following:

```python
from draw_sky2 import draw_sky

n_sky = 3 #choose a file/sky to examine.
data = np.genfromtxt("data/Train_Skies/Train_Skies/\
Training_Sky%d.csv" % (n_sky),
                      dtype = None,
                      skip_header = 1,
                      delimiter = ",",
                      usecols = [1,2,3,4])
print("Data on galaxies in sky %d."%n_sky)
print("position_x, position_y, e_1, e_2 ")
print(data[:3])

fig = draw_sky(data)
plt.title("Galaxy positions and ellipcities of sky %d." % n_sky)
plt.xlabel("x-position")
plt.ylabel("y-position");
```

### Priors

Each sky has one, two or three dark matter halos in it. Tim's solution details that his prior distribution of halo positions was uniform, i.e.

\begin{align}
& x_i \sim \text{Uniform}( 0, 4200)\\\\
& y_i \sim \text{Uniform}( 0, 4200), \;\; i=1,2,3\\\\
\end{align}

Tim and other competitors noted that most skies had one large halo and other halos, if present, were much smaller. Larger halos, having more mass, will influence the surrounding galaxies more. He decided that the large halos would have a mass distributed as a *log*-uniform random variable between 40 and 180 i.e.

$$  m_{\text{large} } = \log \text{Uniform}( 40, 180 ) $$

and in PyMC3,

    exp_mass_large = pm.Uniform("exp_mass_large", 40, 180)
    mass_large = pm.Deterministic("mass_large", np.log(exp_max_large))

(This is what we mean when we say *log*-uniform.) For smaller galaxies, Tim set the mass to be the logarithm of 20. Why did Tim not create a prior for the smaller mass, nor treat it as a unknown? I believe this decision was made to speed up convergence of the algorithm. This is not too restrictive, as by construction the smaller halos have less influence on the galaxies.

Tim logically assumed that the ellipticity of each galaxy is dependent on the position of the halos, the distance between the galaxy and halo, and the mass of the halos. Thus the vector of ellipticity of each galaxy, $\mathbf{e}_i$, are *children* variables of the vector of halo positions $(\mathbf{x},\mathbf{y})$, distance (which we will formalize), and halo masses.

Tim conceived a relationship to connect positions and ellipticity by reading literature and forum posts. He supposed the following was a reasonable relationship:

$$ e_i | ( \mathbf{x}, \mathbf{y} ) \sim \text{Normal}( \sum_{j = \text{halo positions} }d_{i,j} m_j f( r_{i,j} ), \sigma^2 ) $$

where $d_{i,j}$ is the *tangential direction* (the direction in which halo $j$ bends the light of galaxy $i$), $m_j$ is the mass of halo $j$, $f(r_{i,j})$ is a *decreasing function* of the Euclidean distance between halo $j$ and galaxy $i$.

Tim's function $f$ was defined:

$$ f( r_{i,j} ) = \frac{1}{\min( r_{i,j}, 240 ) } $$

for large halos, and for small halos

$$ f( r_{i,j} ) = \frac{1}{\min( r_{i,j}, 70 ) } $$

This fully bridges our observations and unknown. This model is incredibly simple, and Tim mentions this simplicity was purposefully designed: it prevents the model from overfitting.

### Training & PyMC3 implementation

For each sky, we run our Bayesian model to find the posteriors for the halo positions — we ignore the (known) halo position. This is slightly different than perhaps traditional approaches to Kaggle competitions, where this model uses no data from other skies nor the known halo location. That does not mean other data are not necessary — in fact, the model was created by comparing different skies.

```python
import pymc3 as pm
import theano.tensor as T

def euclidean_distance(x, y):
    return np.sqrt(((x - y)**2)).sum(axis=1)

def f_distance(gxy_pos, halo_pos, c):
    # foo_position should be a 2-d numpy array
    # T.maximum() provides our element-wise maximum as in NumPy, but instead for theano tensors
    return T.maximum(euclidean_distance(gxy_pos, halo_pos), c)[:, None]

def tangential_distance(glxy_position, halo_position):
    # foo_position should be a 2-d numpy array
    delta = glxy_position - halo_position
    t = (2*T.arctan(delta[:,1]/delta[:,0]))
    return T.stack([-T.cos(t), -T.sin(t)], axis=1)

with pm.Model() as model:
    #set the size of the halo's mass
    mass_large = pm.Uniform("mass_large", 40, 180)

    #set the initial prior position of the halos, it's a 2-d Uniform dist.
    halo_position = pm.Uniform("halo_position", 0, 4200, shape=(1,2))

    mean = pm.Deterministic("mean", mass_large /\
            f_distance(T.as_tensor(data[:,:2]), halo_position, 240)*\
            tangential_distance(T.as_tensor(data[:,:2]), halo_position))

    ellpty = pm.Normal("ellipcity", mu=mean, tau=1./0.05, observed=data[:,2:])
```

```python
with model:
    # mu, sds, elbo = pm.variational.advi(n=50000)
    trace = pm.sample(5000, init='advi', n_init=50000)
#     step = pm.NUTS(scaling=model.dict_to_array(sds), is_cov=True)
#     trace = pm.sample(5000, step=step, start=mu)
```

We use ADVI here to find a good starting point and scaling for our NUTS sampler. NUTS is a "smarter" MCMC sampling method than Metropolis, so as a result we need fewer total samples for our chains to converge. ADVI follows a completely different methodology to fit a model that we will not get into here. We may cover ADVI, as well as NUTS, in-depth in a later chapter.

Below we plot a "heatmap" of the posterior distribution. (Which is just a scatter plot of the posterior, but we can visualize it as a heatmap.)

```python
t = trace["halo_position"].reshape(20000,2)

fig = draw_sky(data)
plt.title("Galaxy positions and ellipcities of sky %d." % n_sky)
plt.xlabel("x-position")
plt.ylabel("y-position")
plt.scatter(t[:,0], t[:,1], alpha = 0.015, c = "r")
plt.xlim(0, 4200)
plt.ylim(0, 4200);
```

The most probable position reveals itself like a lethal wound.

Associated with each sky is another data point, located in `./data/Training_halos.csv` that holds the locations of up to three dark matter halos contained in the sky. For example, the night sky we trained on has halo locations:

```python
halo_data = np.genfromtxt("data/Training_halos.csv",
                          delimiter = ",",
                          usecols = [1, 2, 3, 4, 5, 6, 7, 8, 9],
                          skip_header = 1)
print(halo_data[n_sky])
```

The third and fourth column represent the true $x$ and $y$ position of the halo. It appears that the Bayesian method has located the halo within a tight vicinity.

```python
fig = draw_sky(data)
plt.title("Galaxy positions and ellipcities of sky %d." % n_sky)
plt.xlabel("x-position")
plt.ylabel("y-position")
plt.scatter(t[:,0], t[:,1], alpha = 0.015, c = "r")
plt.scatter(halo_data[n_sky-1][3], halo_data[n_sky-1][4],
            label = "True halo position",
            c = "k", s = 70)
plt.legend(scatterpoints = 1, loc = "lower left")
plt.xlim(0, 4200)
plt.ylim(0, 4200);

print("True halo location:", halo_data[n_sky][3], halo_data[n_sky][4])
```

Perfect. Our next step is to use the loss function to optimize our location. A naive strategy would be to simply choose the mean:

```python
mean_posterior = t.mean(axis=0).reshape(1,2)
print(mean_posterior)
```

```python
from DarkWorldsMetric import main_score

_halo_data = halo_data[n_sky-1]

nhalo_all =  _halo_data[0].reshape(1,1)
x_true_all = _halo_data[3].reshape(1,1)
y_true_all = _halo_data[4].reshape(1,1)
x_ref_all = _halo_data[1].reshape(1,1)
y_ref_all = _halo_data[2].reshape(1,1)
sky_prediction = mean_posterior

print("Using the mean:")
main_score(nhalo_all, x_true_all, y_true_all, \
            x_ref_all, y_ref_all, sky_prediction)

#what's a bad score?
random_guess = np.random.randint(0, 4200, size=(1,2))
print("Using a random location:", random_guess)
main_score(nhalo_all, x_true_all, y_true_all, \
            x_ref_all, y_ref_all, random_guess)
```

This is a good guess, it is not very far from the true location, but it ignores the loss function that was provided to us. We also need to extend our code to allow for up to two additional, *smaller* halos: Let's create a function for automatizing our PyMC3.

```python
def halo_posteriors(n_halos_in_sky, galaxy_data,samples = 5e5, burn_in = 500):
    #set the size of the halo's mass
    with pm.Model() as model:
        mass_large = pm.Uniform("mass_large", 40, 180)

        mass_small_1 = 20
        mass_small_2 = 20

        masses = np.array([mass_large,mass_small_1, mass_small_2], dtype=object)

        #set the initial prior positions of the halos, it's a 2-d Uniform dist.
        halo_positions = pm.Uniform("halo_positions", 0, 4200, shape=(n_halos_in_sky,2)) #notice this size

        fdist_constants = np.array([240, 70, 70])

        _sum = 0
        for i in range(n_halos_in_sky):
            _sum += masses[i]/f_distance(data[:,:2], halo_positions[i, :], fdist_constants[i])*\
                tangential_distance(data[:,:2], halo_positions[i, :])

        mean = pm.Deterministic("mean", _sum)

        ellpty = pm.Normal("ellipcity", mu=mean, tau=1./0.05, observed=data[:,2:])

        trace = pm.sample(5000, init='advi', n_init=50000)
#         mu, sds, elbo = pm.variational.advi(n=50000)
#         step = pm.NUTS(scaling=model.dict_to_array(sds), is_cov=True)
#         trace = pm.sample(samples, step=step, start=mu)

    burned_trace = trace[burn_in:]
    return burned_trace["halo_positions"]
```

```python
n_sky = 215
data = np.genfromtxt("data/Train_Skies/Train_Skies/\
Training_Sky%d.csv" % (n_sky),
                      dtype = None,
                      skip_header = 1,
                      delimiter = ",",
                      usecols = [1,2,3,4])
```

```python
#there are 3 halos in this file.
samples = 5000
traces = halo_posteriors(3, data, samples = samples, burn_in=500)
```

```python
fig = draw_sky(data)
plt.title("Galaxy positions and ellipcities of sky %d." % n_sky)
plt.xlabel("x-position")
plt.ylabel("y-position")

colors = ["#467821", "#A60628", "#7A68A6"]

for i in range(traces.shape[1]):
    plt.scatter(traces[:, i, 0],  traces[:, i, 1], c = colors[i], alpha = 0.02)

for i in range(traces.shape[1]):
    plt.scatter(halo_data[n_sky-1][3 + 2*i], halo_data[n_sky-1][4 + 2*i],
            label = "True halo position", c = "k", s = 90)

#plt.legend(scatterpoints = 1)
plt.xlim(0, 4200)
plt.ylim(0, 4200);
```

This looks pretty good, though it took a long time for the system to (sort of) converge. Our optimization step would look something like this:

```python
_halo_data = halo_data[n_sky-1]
print(traces.shape)

mean_posterior = traces.mean(axis=0).reshape(1,6)
print(mean_posterior)

nhalo_all =  _halo_data[0].reshape(1,1)
x_true_all = _halo_data[3].reshape(1,1)
y_true_all = _halo_data[4].reshape(1,1)
x_ref_all = _halo_data[1].reshape(1,1)
y_ref_all = _halo_data[2].reshape(1,1)
sky_prediction = mean_posterior

print("Using the mean:")
main_score([1], x_true_all, y_true_all, \
            x_ref_all, y_ref_all, sky_prediction)

#what's a bad score?
random_guess = np.random.randint(0, 4200, size=(1,2))
print("Using a random location:", random_guess)
main_score([1], x_true_all, y_true_all, \
            x_ref_all, y_ref_all, random_guess)
```

## References
1. Antifragile: Things That Gain from Disorder. New York: Random House. 2012. ISBN 978-1-4000-6782-4.
1.  [Tim Saliman's solution to the Dark World's Contest](http://www.timsalimans.com/observing-dark-worlds)
2. Silver, Nate. The Signal and the Noise: Why So Many Predictions Fail — but Some Don't. 1. Penguin Press HC, The, 2012. Print.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```


```python
import matplotlib.pyplot as plt
import seaborn as sns
import itertools
import numpy as np
from scipy.stats import binom, norm
from scipy import integrate
from collections import namedtuple
from matplotlib import cm
import pandas as pd
import six
if six.PY3:
    from importlib import reload
import luigi
import pickle
from pprint import pprint

%matplotlib inline
%load_ext autoreload
%autoreload 2
```

# k-NN, Function Expectation, Density Estimation

```python
from experiment_framework.helpers import build_convergence_curve_pipeline
from empirical_privacy.one_bit_sum import GenSampleOneBitSum

# from empirical_privacy import one_bit_sum_joblib as one_bit_sum
# from empirical_privacy import lsdd
# reload(one_bit_sum)
```

```python

def B_pmf(k, n, p):
    return binom(n, p).pmf(k)
def B0_pmf(k, n, p):
    return B_pmf(k, n-1, p)
def B1_pmf(k, n, p):
    return B_pmf(k-1, n-1, p)
def sd(N, P):
    return 0.5*np.sum(abs(B0_pmf(i, N, P) - B1_pmf(i, N, P)) for i in range(N+1))
def optimal_correctness(n, p):
    return 0.5 + 0.5*sd(n, p)
```

```python

```

```python
n_max = 2**10
ntri=30
n=7
p=0.5
sd(n,p)
```

    0.31250000000000022

```python
B0 = [B0_pmf(i, n, p) for i in range(n+1)]
B1 = [B1_pmf(i, n, p) for i in range(n+1)]
dif = np.abs(np.array(B0)-np.array(B1))
sdv = 0.5*np.sum(dif)
pc = 0.5+0.5*sdv
print(f'n={n} coin flips p={p} probability of heads'\
      '\nB0 has first outcome=0, B1 has first outcome=1')
print(f'Statistic is the total number of heads sum')
print(f'N_heads=\t{"     ".join(np.arange(n+1).astype(str))}')
print(f'PMF of B0=\t{B0}\nPMF of B1=\t{B1}')
print(f'|B0-B1|=\t{dif}')
print(f'sd = 0.5 * sum(|B0-B1|) = {sdv}')
print(f'P(Correct) = 0.5 + 0.5*sd = {pc}')
```

    n=7 coin flips p=0.5 probability of heads
    B0 has first outcome=0, B1 has first outcome=1
    Statistic is the total number of heads sum
    N_heads=	0     1     2     3     4     5     6     7
    PMF of B0=	[0.015625000000000007, 0.093750000000000028, 0.23437500000000003, 0.31250000000000022, 0.23437500000000003, 0.093750000000000028, 0.015625000000000007, 0.0]
    PMF of B1=	[0.0, 0.015625000000000007, 0.093750000000000028, 0.23437500000000003, 0.31250000000000022, 0.23437500000000003, 0.093750000000000028, 0.015625000000000007]
    |B0-B1|=	[ 0.015625  0.078125  0.140625  0.078125  0.078125  0.140625  0.078125
      0.015625]
    sd = 0.5 * sum(|B0-B1|) = 0.3125000000000002
    P(Correct) = 0.5 + 0.5*sd = 0.6562500000000001

```python
ccc_kwargs = {
    'confidence_interval_width':10,
    'n_max':2**13,
    'dataset_settings' : {
        'n_trials':n,
        'prob_success':p,
        'gen_distr_type':'binom'
    },
    'validation_set_size' : 2000
}

CCCs = []
Fits = ['knn', 'density', 'expectation']
for fit in Fits:
    CCCs.append(build_convergence_curve_pipeline(
                    GenSampleOneBitSum,
        gensample_kwargs = {'generate_in_batch':True},
        fitter=fit,
        fitter_kwargs={} if fit=='knn' else {'statistic_column':0}
                )(**ccc_kwargs)
   )
luigi.build(CCCs, local_scheduler=True, workers=4, log_level='ERROR')

colors = cm.Accent(np.linspace(0,1,len(CCCs)+1))

ax = plt.figure(figsize=(10,5))
ax = plt.gca()
leg_handles = []
for (i, CC) in enumerate(CCCs):
    with CC.output().open() as f:
        res = pickle.load(f)
    handle=sns.tsplot(res['sd_matrix'], ci='sd', color=colors[i], ax=ax, legend=False, time=res['training_set_sizes'])

j=0
for i in range(len(CCCs), 2*len(CCCs)):
    handle.get_children()[i].set_label('{}'.format(Fits[j]))
    j+=1
plt.semilogx()
plt.axhline(optimal_correctness(n, p), linestyle='--', color='r', label='_nolegend_')
plt.axhline(0.5, linestyle='-', color='b', label='_nolegend_')
plt.title('n={n} p={p} $\delta$={d:.3f}'.format(n=n, p=p, d=sd(n,p)), fontsize=20)
plt.xlabel('num samples')
plt.ylabel('Correctness Rate')
plt.legend(loc=(0,1.1))

```

### Repeat the above using joblib to make sure the luigi implementation is correct

```python
from math import ceil, log
one_bit_sum.n_jobs=1
N = int(ceil(log(n_max) / log(2)))
N_samples = np.logspace(4,N,num=N-3, base=2).astype(np.int)

ax = plt.figure(figsize=(10,5))
ax = plt.gca()
AlgArg = namedtuple('AlgArg', field_names=['f_handle', 'f_kwargs'])
algs = [
    AlgArg(one_bit_sum.get_knn_correctness_rate_cached, {'neighbor_method':'sqrt'}),
    AlgArg(one_bit_sum.get_knn_correctness_rate_cached, {'neighbor_method':'sqrt_random_tiebreak'}),

    AlgArg(one_bit_sum.get_density_est_correctness_rate_cached, {'bandwidth_method':None}),
    AlgArg(one_bit_sum.get_expectation_correctness_rate_cached, {'bandwidth_method':None}),
    AlgArg(one_bit_sum.get_lsdd_correctness_rate_cached, {})
    #AlgArg(one_bit_sum.get_knn_correctness_rate_cached, {'neighbor_method':'cv'})
]
colors = cm.Accent(np.linspace(0,1,len(algs)+1))
leg_handles = []
for (i,alg) in enumerate(algs):
    res = one_bit_sum.get_res(n,p,ntri, alg.f_handle, alg.f_kwargs, n_max=n_max)
    handle=sns.tsplot(res, ci='sd', color=colors[i], ax=ax, legend=False, time=N_samples)
#     f, coef = get_fit(res, N_samples)
#     print alg, coef
#     lim = coef[0]
#     plt.plot(N_samples, f(N_samples), linewidth=3)
#    plt.text(N_samples[-1], lim, '{:.3f}'.format(lim),fontsize=16)
j=0
for i in range(len(algs), 2*len(algs)):
    #print i, i/2-1 if i%2==0 else (i)/2
    handle.get_children()[i].set_label('{} {}'.format(algs[j].f_handle.func.__name__, algs[j].f_kwargs))
    j+=1
plt.semilogx()
plt.axhline(optimal_correctness(n, p), linestyle='--', color='r', label='_nolegend_')
plt.axhline(0.5, linestyle='-', color='b', label='_nolegend_')
plt.title('n={n} p={p} $\delta$={d:.3f}'.format(n=n, p=p, d=sd(n,p)), fontsize=20)
plt.xlabel('num samples')
plt.ylabel('Correctness Rate')
plt.legend(loc=(0,1.1))
#print ax.get_legend_handles_labels()
```

### Timing GenSamples

Without halving: 7.5sec
With halving: 8.1sec (i.e. not much overhead)

```python
from luigi_utils.sampling_framework import GenSamples
import time

class GS(GenSamples(GenSampleOneBitSum, generate_in_batch=True)):
    pass
GSi = GS(dataset_settings = ccc_kwargs['dataset_settings'],
  random_seed='0',
  generate_positive_samples=True,
  num_samples=2**15)

start = time.time()
luigi.build([GSi], local_scheduler=True, workers=8, log_level='ERROR')
cputime = time.time() - start
print(cputime)
```

    8.109845638275146

```python
res['training_set_sizes'].shape
```

    (8,)

```python
np.concatenate((np.array([]), np.array([1,2,3])))
```

    array([ 1.,  2.,  3.])

### More exp

```python
def get_fit(res, N_samples):
    ntri, nsamp = res.shape
    sqrt2 = np.sqrt(2)
    Xlsq = np.hstack((np.ones((nsamp,1)),
                      sqrt2/(N_samples.astype(np.float)**0.25)[:, np.newaxis]))
    y = 1.0 - res.reshape((nsamp*ntri, 1))

    Xlsq = reduce(lambda x,y: np.vstack((x,y)), [Xlsq]*ntri)
    coef = np.linalg.lstsq(Xlsq, y)[0].ravel()
    f = lambda n: 1.0 - coef[0] - coef[1]*sqrt2/n.astype(np.float)**0.25, coef
    return f

```

```python
trial=0
num_samples=2**11
bandwidth_method=None
from scipy.stats import gaussian_kde

X0, X1, y0, y1 = one_bit_sum.gen_data(n, p, num_samples, trial)
X0 = X0.ravel()
X1 = X1.ravel()
bw = None
if hasattr(bandwidth_method, '__call__'):
    bw = float(bandwidth_method(num_samples)) / num_samples  # eg log
if type(bandwidth_method) == float:
    bw = num_samples**(1-bandwidth_method)
f0 = gaussian_kde(X0, bw_method = bw)
f1 = gaussian_kde(X1, bw_method = bw)
#Omega = np.unique(np.concatenate((X0, X1)))
_min = 0
_max = n
x = np.linspace(_min, _max, num=10*num_samples)
print('difference of densities=',0.5 + 0.5 * 0.5 * np.mean(np.abs(f0(x)-f1(x))))
denom = f0(x)+f1(x)
numer = np.abs(f0(x)-f1(x))
print('expectation = ',0.5 + 0.5*np.mean(numer/denom))
```

# Uniforml distributed random variables

$$g_0 = U[0,0.5]+\sum_{i=1}^{n-1} U[0,1]$$

$$g_1 = U[0.5,1.0]+\sum_{i=1}^{n-1} U[0,1]$$

Let $\mu_n = \frac{n-1}{2}$ and $\sigma_n = \sqrt{\frac{n-0.75}{12}}$

By the CLT $g_0\sim N(\mu_n+0.25, \sigma_n)$ and $g_1\sim N(\mu_n+0.75, \sigma_n)$.

```python
from math import sqrt
n=3
x = np.linspace(n/2.0-sqrt(n), n/2.0+sqrt(n))
sigma = sqrt((n-0.75)/12.0)
sqrt2 = sqrt(2)
mu = (n-1.0)/2

def g0_pdf(x):
    return norm.pdf(x, loc=mu+0.25, scale=sigma)
def g1_pdf(x):
    return norm.pdf(x, loc=mu+0.75, scale=sigma)
def d_pdf(x):
    return norm.pdf(x, loc=-0.5, scale=sigma*sqrt2)
def g_int(n):
    sigma = sqrt((n-0.75)/12.0)
    mu = (n-1.0)/2
    N0 = norm(loc=mu+0.25, scale=sigma)
    N1 = norm(loc=mu+0.75, scale=sigma)
    I0 = N0.cdf(n*0.5)-N0.cdf(0)
    I1 = N1.cdf(n*0.5)-N1.cdf(0)
    return 2*(I0-I1)
def g_stat_dist(n):
    return 0.5 * g_int(n)
def g_optimal_correctness(n):
    return 0.5 + 0.5*g_stat_dist(n)

plt.plot(x, g0_pdf(x), label='$g_0$')
plt.plot(x, g1_pdf(x), label='$g_1$')
#plt.plot(x, d_pdf(x), label='$d$')
plt.axvline(x=n/2.0, color='r')
assert g0_pdf(n/2.0)==g1_pdf(n/2.0)
plt.legend()
print(g_optimal_correctness(n))
```

```python
from math import ceil, log
if n_max >= 2**13:
    one_bit_sum.n_jobs=1
else:
    one_bit_sum.n_jobs=-1

N = int(ceil(log(n_max) / log(2)))
N_samples = np.logspace(4,N,num=N-3, base=2).astype(np.int)

ax = plt.figure(figsize=(10,5))
ax = plt.gca()
AlgArg = namedtuple('AlgArg', field_names=['f_handle', 'f_kwargs'])
algs = [
    AlgArg(one_bit_sum.get_knn_correctness_rate_cached, {'neighbor_method':'sqrt'}),
    AlgArg(one_bit_sum.get_knn_correctness_rate_cached, {'neighbor_method':'sqrt_random_tiebreak'}),

    AlgArg(one_bit_sum.get_density_est_correctness_rate_cached, {'bandwidth_method':None}),
    AlgArg(one_bit_sum.get_expectation_correctness_rate_cached, {'bandwidth_method':None}),
    AlgArg(one_bit_sum.get_lsdd_correctness_rate_cached, {})
    #AlgArg(one_bit_sum.get_knn_correctness_rate_cached, {'neighbor_method':'cv'})
]
for A in algs:
    A.f_kwargs['type']='norm'
colors = cm.Accent(np.linspace(0,1,len(algs)+1))
leg_handles = []
for (i,alg) in enumerate(algs):
    res = one_bit_sum.get_res(n,p,ntri, alg.f_handle, alg.f_kwargs, n_max=n_max)
    handle=sns.tsplot(res, ci='sd', color=colors[i], ax=ax, legend=False, time=N_samples)
#     f, coef = get_fit(res, N_samples)
#     print alg, coef
#     lim = coef[0]
#     plt.plot(N_samples, f(N_samples), linewidth=3)
#     plt.text(N_samples[-1], lim, '{:.3f}'.format(lim),fontsize=16)
j=0
for i in range(len(algs), 2*len(algs)):
    #print i, i/2-1 if i%2==0 else (i)/2
    handle.get_children()[i].set_label(algs[j].f_handle.func.__name__)
    j+=1
    #print handle.get_children()[i].get_label()
plt.semilogx()
plt.axhline(g_optimal_correctness(n), linestyle='--', color='r', label='_nolegend_')
plt.axhline(0.5, linestyle='-', color='b', label='_nolegend_')
plt.title('n={n} $\delta$={d:.3f}'.format(n=n, d=g_stat_dist(n)), fontsize=20)
plt.xlabel('num samples')
plt.ylabel('Correctness Rate')
plt.legend(loc=(1.1,0))
#print ax.get_legend_handles_labels()
```

```python
true_value = g_optimal_correctness(n)
print(true_value)
```

```python
trial=0
num_samples=2**15
bandwidth_method=None
from scipy.stats import gaussian_kde

X0, X1, y0, y1 = one_bit_sum.gen_data(n, p, num_samples, trial, type='norm')
X0 = X0.ravel()
X1 = X1.ravel()
bw = None
if hasattr(bandwidth_method, '__call__'):
    bw = float(bandwidth_method(num_samples)) / num_samples  # eg log
if type(bandwidth_method) == float:
    bw = num_samples**(1-bandwidth_method)
f0 = gaussian_kde(X0, bw_method = bw)
f1 = gaussian_kde(X1, bw_method = bw)
#Omega = np.unique(np.concatenate((X0, X1)))
_min = 0
_max = n
x = np.linspace(_min, _max, num=num_samples)

```

```python
print('difference of densities=',0.5 + 0.5 * 0.5 * integrate.quad(lambda x: np.abs(f0(x)-f1(x)), -np.inf, np.inf)[0])
```

```python
X = np.concatenate((X0,X1))
f0x = f0(X)
f1x = f1(X)
denom = (f0x+f1x+np.spacing(1))
numer = np.abs(f0x-f1x)
print('expectation = ',0.5 + 0.5*np.mean(numer/denom))
```

```python
print('exact=',g_optimal_correctness(n))
```

```python
plt.plot(x, f0(x),label='$\hat g_0$', linestyle='--')
plt.plot(x, f1(x),label='$\hat g_1$', linestyle='--')
plt.plot(x, g0_pdf(x), label='$g_0$')
plt.plot(x, g1_pdf(x), label='$g_1$')
plt.legend(loc=(1.05,0))
```

### Comparing different numerical integration techniques

```python
to_int = [f0,f1]

print 'Quad'
# for (i,f) in enumerate(to_int):
#     intr = integrate.quad(f, -np.inf, np.inf)
#     print 'func={0} err={1:.3e}'.format(i, abs(1-intr[0]))
g_int(n)-integrate.quad(lambda x: np.abs(f0(x)-f1(x)), -np.inf, np.inf)[0]
```

```python
to_int = [f0,f1]

print 'Quad'
g_int(n)-integrate.quad(lambda x: np.abs(f0(x)-f1(x)), -np.inf, np.inf)[0]
```

```python
g_int(n)
```

```python
print 'Simps'
def delta(x):
    return np.abs(f0(x)-f1(x))
X = np.unique(np.concatenate((X0,X1)))
y = delta(X)
g_int(n)-integrate.simps(y,X)
```

```python
import empirical_privacy.lsdd
```

```python
rtv = lsdd.lsdd(X0[np.newaxis, :], X1[np.newaxis, :])
```

```python
plt.hist(rtv[1])
```

```python
np.mean(rtv[1])
```

## Sympy-based analysis

```python
import sympy as sy
n,k = sy.symbols('n k', integer=True)
#k = sy.Integer(k)
p = sy.symbols('p', real=True)
q=1-p

def binom_pmf(k, n, p):
    return sy.binomial(n,k)*(p**k)*(q**(n-k))
def binom_cdf(x, n, p):
    return sy.Sum([binom_pmf(j, n, p) for j in sy.Range(x+1)])

B0 = binom_pmf(k, n-1, p)
B1 = binom_pmf(k-1, n-1, p)
```

```python
def stat_dist(N,P):
    return 0.5*sum([sy.Abs(B0.subs([(n,N),(p,P), (k,i)])-B1.subs([(n,N),(p,P), (k,i)])) for i in range(N+1)])
def sd(N, P):
    return 0.5*np.sum(abs(B0(i, N, P) - B1(i, N, P)) for i in range(N+1))
```

```python
stat_dist(50,0.5)
```

```python
sd(5000,0.5)
```

```python
N=2
terms =[(B0.subs([(n,N), (k,i)]).simplify(),B1.subs([(n,N), (k,i)]).simplify()) for i in range(N+1)]
print terms
```

```python
0.5*sum(map(lambda t: sy.Abs(t[0]-t[1]), terms)).subs([(p,0.5)])
```

```python
stat_dist(4,0.5)
```


```python
from IPython.display import HTML
tag = HTML('''
Promijeni vidljivost <a href="javascript:code_toggle()">ovdje</a>.''')
display(tag)
```

Promijeni vidljivost ovdje.

```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

%matplotlib notebook
import numpy as np
import math
import matplotlib.pyplot as plt
from scipy import signal
import ipywidgets as widgets
import control as c
import sympy as sym
from IPython.display import Latex, display, Markdown # For displaying Markdown and LaTeX code
from fractions import Fraction
import matplotlib.patches as patches
```

## Aproksimacija dominantnim polom

Pri proučavanju ponašanja sustava, često ih se aproksimira dominantnim polom ili parom dominantnih kompleksnih polova. Ovaj primjer demonstrira navedeno svojstvo.

Sustav drugog reda definiran je sljedećom prijenosnom funkcijom:

\begin{equation}
    G(s)=\frac{\alpha\beta}{(s+\alpha)(s+\beta)}=\frac{1}{(\frac{1}{\alpha}s+1)(\frac{1}{\beta}s+1)},
\end{equation}

gdje je $\beta=1$, a $\alpha$ je iterabilan.

Sustav trećeg reda definiran je sljedećom prijenosnom funkcijom:

\begin{equation}
    G(s)=\frac{\alpha{\omega_0}^2}{\big(s+\alpha\big)\big(s^2+2\zeta\omega_0s+\omega_0^2\big)}=\frac{1}{(\frac{1}{\alpha}s+1)(\frac{1}{\omega_0^2}s^2+\frac{2\zeta\alpha}{\omega_0}s+1)},
\end{equation}

gdje je $\beta=1$, $\omega_0=4.1$, $\zeta=0.24$ i $\alpha$ je iterabilan.

---

### Kako koristiti ovaj interaktivni primjer?

Odaberite između sustava drugog i trećeg reda te pomičite klizač za promjenu položaja pomičnog pola $a$.

Ovaj interaktivni primjer zasnovan je na sljedećem [tutorijalu](https://lpsa.swarthmore.edu/PZXferStepBode/DomPole.html "The Dominant Pole Approximation") autora Prof. Erika Cheevera.

```python
# System selector buttons
style = {'description_width': 'initial'}
typeSelect = widgets.ToggleButtons(
    options=[('sustav drugog reda', 0), ('sustav trećeg reda', 1),],
    description='Select: ',style=style)
```

```python
display(typeSelect)
continuous_update=False

# set up plot

fig, ax = plt.subplots(2,1,figsize=[9.8,7],num='Aproksimacija dominantnim polom')
plt.subplots_adjust(hspace=0.35)
ax[0].grid(True)
ax[1].grid(True)
# ax[2].grid(which='both', axis='both', color='lightgray')
ax[0].axhline(y=0,color='k',lw=.8)
ax[1].axhline(y=0,color='k',lw=.8)
ax[0].axvline(x=0,color='k',lw=.8)
ax[1].axvline(x=0,color='k',lw=.8)
ax[0].set_xlabel('Re')
ax[0].set_ylabel('Im')
ax[0].set_xlim([-10,0.5])
ax[1].set_xlim([-0.5,20])
ax[1].set_xlabel('$t$ [s]')
ax[1].set_ylabel('ulaz, izlaz')
ax[0].set_title('Dijagram polova i nula')
ax[1].set_title('Vremenski odziv')

plotzero, = ax[0].plot([], [])
response, = ax[1].plot([], [])
responseAdom, = ax[1].plot([], [])
responseBdom, = ax[1].plot([], [])

ax[1].step([0,50],[0,1],color='C0',label='ulaz')

# generate x values

def response_func(a,index):

    global plotzero, response, responseAdom, responseBdom
#     global bodePlot, bodePlotAdom, bodePlotBdom

    t = np.linspace(0, 50, 1000)

    if index==0:
        b=1
        num=a*b
        den=([1,a+b,a*b])
        tf_sys=c.TransferFunction(num,den)
        poles_sys,zeros_sys=c.pzmap(tf_sys, Plot=False)
        tout, yout = c.step_response(tf_sys,t)
        den1=([1,a])
        tf_sys1=c.TransferFunction(a,den1)
        toutA, youtA = c.step_response(tf_sys1,t)
        den2=([1,b])
        tf_sys2=c.TransferFunction(b,den2)
        toutB, youtB = c.step_response(tf_sys2,t)
        mag, phase, omega = c.bode_plot(tf_sys, Plot=False)   # Bode-plot
        magA, phase, omegaA = c.bode_plot(tf_sys1, Plot=False)   # Bode-plot
        magB, phase, omegaB = c.bode_plot(tf_sys2, Plot=False)   # Bode-plot
        s=sym.Symbol('s')
        eq=(a*b/((s+a)*(s+b)))
        eq1=1/(((1/a)*s+1)*((1/b)*s+1))
        display(Markdown('Pomični pol (ljubičasta krivulja) $\\alpha$ je jednak %.1f, fiksni pol (crvena krivulja) $b$ je jednak %i; Prijenosna funkcija je:'%(a,1)))
        display(eq),display(Markdown('or')),display(eq1)

    elif index==1:
        omega0=4.1
        zeta=0.24
        num=a*omega0**2
        den=([1,2*zeta*omega0+a,omega0**2+2*zeta*omega0*a,a*omega0**2])
        tf_sys=c.TransferFunction(num,den)
        poles_sys,zeros_sys=c.pzmap(tf_sys, Plot=False)
        tout, yout = c.step_response(tf_sys,t)
        den1=([1,a])
        tf_sys1=c.TransferFunction(a,den1)
        toutA, youtA = c.step_response(tf_sys1,t)
        den2=([1,2*zeta*omega0,omega0**2])
        tf_sys2=c.TransferFunction(omega0**2,den2)
        toutB, youtB = c.step_response(tf_sys2,t)
        mag, phase, omega = c.bode_plot(tf_sys, Plot=False)   # Bode-plot
        magA, phase, omegaA = c.bode_plot(tf_sys1, Plot=False)   # Bode-plot
        magB, phase, omegaB = c.bode_plot(tf_sys2, Plot=False)   # Bode-plot
        s=sym.Symbol('s')
        eq=(a*omega0**2/((s+a)*(s**2+2*zeta*omega0*s+omega0*omega0)))
        eq1=1/(((1/a)*s+1)*((1/(omega0*omega0))*s*s+(2*zeta*a/omega0)*s+1))

        display(Markdown('Pomični pol (ljubičasta krivulja) $\\alpha$ je jednak %.1f, fiksni pol (crvena krivulja) $\\beta$ je jednak $1\pm4j$ ($\omega_0$ je postavljen na 4.1, $\zeta$ je postavljen na 0.24). Prijenosna funkcija je:'%(a)))
        display(eq),display(Markdown('ili')),display(eq1)

    ax[0].lines.remove(plotzero)
    ax[1].lines.remove(response)
    ax[1].lines.remove(responseAdom)
    ax[1].lines.remove(responseBdom)

    plotzero, = ax[0].plot(np.real(poles_sys), np.imag(poles_sys), 'xg', markersize=10, label = 'pol')
    response, = ax[1].plot(tout,yout,color='C1',label='odziv sustava',lw=3)
    responseAdom, = ax[1].plot(toutA,youtA,color='C4',label='odziv zasnovan samo na pomičnom polu (paru polova)')
    responseBdom, = ax[1].plot(toutB,youtB,color='C3',label='odziv zasnovan samo na fiksnom polu')

    ax[0].legend()
    ax[1].legend()

a_slider=widgets.FloatSlider(value=0.1, min=0.1, max=10, step=.1,
                                        description='$\\alpha$:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)

input_data=widgets.interactive_output(response_func,{'a':a_slider,'index':typeSelect})

def update_slider(index):
    global a_slider

    aval=[0.1,0.1]
    a_slider.value=aval[index]

input_data2=widgets.interactive_output(update_slider,{'index':typeSelect})

display(a_slider,input_data)

```

    ToggleButtons(description='Select: ', options=(('sustav drugog reda', 0), ('sustav trećeg reda', 1)), style=To…

    <IPython.core.display.Javascript object>

    FloatSlider(value=0.1, continuous_update=False, description='$\\alpha$:', max=10.0, min=0.1)

    Output()

```python

```


[Sebastian Raschka](http://sebastianraschka.com)

- [Open in IPython nbviewer](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/python_howtos/scikit_linear_classificationreate=1)

- [Link to this IPython notebook on Github](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/scikit-learn/scikit_linear_classification.ipynb)

- [Link to the GitHub Repository pattern_classification](https://github.com/rasbt/pattern_classification)

```python
%load_ext watermark
%watermark -a 'Gopala KR' -u -d -v -p numpy,scikit-learn,matplotlib
```

    The watermark extension is already loaded. To reload it, use:
      %reload_ext watermark
    Gopala KR
    last updated: 2018-01-28

    CPython 3.6.3
    IPython 6.2.1

    numpy 1.13.1
    scikit-learn 0.19.0
    matplotlib 2.0.2

    /srv/venv/lib/python3.6/site-packages/watermark/watermark.py:155: DeprecationWarning: Importing scikit-learn as `scikit-learn` has been depracated and will not be supported anymore in v1.7.0. Please use the package name `sklearn` instead.
      DeprecationWarning)

I would be happy to hear your comments and suggestions.
Please feel free to drop me a note via
[twitter](https://twitter.com/rasbt), [email](mailto:bluewoodtree@gmail.com), or [google+](https://plus.google.com/+SebastianRaschka).

# An Introduction to simple linear supervised classification using `scikit-learn`

In this introduction I want to give a brief overview of how Python's `scikit-learn` machine learning library can be used for simple linear classification.

# Sections

- [About the dataset](#About-the-dataset)
- [Reading in a dataset from a CSV file](#Reading-in-a-dataset-from-a-CSV-file)
- [Visualizing the Wine dataset](#Visualizing-the-Wine-dataset)
- [Splitting into training and test dataset](#Splitting-into-training-and-test-dataset)
- [Feature Scaling](#Feature-Scaling)
- [Introduction to Multiple Discriminant Analysis (MDA)](#MDA)
- [Classification via LDA](#LDA)
- [Stochastic Gradient Descent (SGD) as linear classifier](#SGD)

## About the dataset

[[back to top]](#Sections)

For the following tutorial, we will be working with the free "Wine" Dataset that is deposited on the UCI machine learning repository
(http://archive.ics.uci.edu/ml/datasets/Wine).

**Reference:**
Forina, M. et al, PARVUS - An Extendible Package for Data
Exploration, Classification and Correlation. Institute of Pharmaceutical
and Food Analysis and Technologies, Via Brigata Salerno,
16147 Genoa, Italy.

The Wine dataset consists of 3 different classes where each row correspond to a particular wine sample.

The class labels (1, 2, 3) are listed in the first column, and the columns 2-14 correspond to the following 13 attributes (features):

1) Alcohol
2) Malic acid
3) Ash
4) Alcalinity of ash
5) Magnesium
6) Total phenols
7) Flavanoids
8) Nonflavanoid phenols
9) Proanthocyanins
10) Color intensity
11) Hue
12) OD280/OD315 of diluted wines
13) Proline

An excerpt from the wine_data.csv dataset:

1,14.23,1.71,2.43,15.6,127,2.8,3.06,.28,2.29,5.64,1.04,3.92,1065
1,13.2,1.78,2.14,11.2,100,2.65,2.76,.26,1.28,4.38,1.05,3.4,1050
[...]
2,12.37,.94,1.36,10.6,88,1.98,.57,.28,.42,1.95,1.05,1.82,520
2,12.33,1.1,2.28,16,101,2.05,1.09,.63,.41,3.27,1.25,1.67,680
[...]
3,12.86,1.35,2.32,18,122,1.51,1.25,.21,.94,4.1,.76,1.29,630
3,12.88,2.99,2.4,20,104,1.3,1.22,.24,.83,5.4,.74,1.42,530

## Reading in a dataset from a CSV file

[[back to top]](#Sections)

Since it is quite typical to have the input data stored locally, as mentioned above, we will use the [`numpy.loadtxt`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html) function  now to read in the data from the CSV file.
(alternatively [`np.genfromtxt()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html) could be used in similar way, it provides some additional options)

```python
import numpy as np

# reading in all data into a NumPy array
all_data = np.loadtxt(open("../../data/wine_data.csv","r"),
        delimiter=",",
        skiprows=0,
        dtype=np.float64
        )

# load class labels from column 1
y_wine = all_data[:,0]

# conversion of the class labels to integer-type array
y_wine = y_wine.astype(np.int64, copy=False)

# load the 14 features
X_wine = all_data[:,1:]

# printing some general information about the data
print('\ntotal number of samples (rows):', X_wine.shape[0])
print('total number of features (columns):', X_wine.shape[1])

# printing the 1st wine sample
float_formatter = lambda x: '{:.2f}'.format(x)
np.set_printoptions(formatter={'float_kind':float_formatter})
print('\n1st sample (i.e., 1st row):\nClass label: {:d}\n{:}\n'
          .format(int(y_wine[0]), X_wine[0]))

# printing the rel.frequency of the class labels
print('Class label frequencies')
print('Class 1 samples: {:.2%}'.format(list(y_wine).count(1)/y_wine.shape[0]))
print('Class 2 samples: {:.2%}'.format(list(y_wine).count(2)/y_wine.shape[0]))
print('Class 3 samples: {:.2%}'.format(list(y_wine).count(3)/y_wine.shape[0]))
```

    total number of samples (rows): 178
    total number of features (columns): 13

    1st sample (i.e., 1st row):
    Class label: 1
    [14.23 1.71 2.43 15.60 127.00 2.80 3.06 0.28 2.29 5.64 1.04 3.92 1065.00]

    Class label frequencies
    Class 1 samples: 33.15%
    Class 2 samples: 39.89%
    Class 3 samples: 26.97%

## Visualizing the Wine dataset

[[back to top]](#Sections)

There are endless way to visualize datasets for get an initial idea of how the data looks like. The most common ones are probably histograms and scatter plots.

Scatter plots are useful for visualizing features in more than just one dimension, for example to get a feeling for the correlation between particular features.
Unfortunately, we can't plot all 13 features here at once, since the visual cortex of us humans is limited to a maximum of three dimensions.

Below, we will create an example 2D-Scatter plot from the features "Alcohol content" and "Malic acid content".
Additionally, we will use the [`scipy.stats.pearsonr`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html) function to calculate a Pearson correlation coefficient between these two features.

```python
%matplotlib inline
```

```python
from matplotlib import pyplot as plt
from scipy.stats import pearsonr

plt.figure(figsize=(10,8))

for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue', 'red', 'green')):

    # Calculate Pearson correlation coefficient
    R = pearsonr(X_wine[:,0][y_wine == label], X_wine[:,1][y_wine == label])
    plt.scatter(x=X_wine[:,0][y_wine == label], # x-axis: feat. from col. 1
                y=X_wine[:,1][y_wine == label], # y-axis: feat. from col. 2
                marker=marker, # data point symbol for the scatter plot
                color=color,
                alpha=0.7,
                label='class {:}, R={:.2f}'.format(label, R[0]) # label for the legend
                )

plt.title('Wine Dataset')
plt.xlabel('alcohol by volume in percent')
plt.ylabel('malic acid in g/l')
plt.legend(loc='upper right')

plt.show()
```

If we want to pack 3 different features into one scatter plot at once, we can also do the same thing in 3D:

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111, projection='3d')

for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue','red','green')):

    ax.scatter(X_wine[:,0][y_wine == label],
               X_wine[:,1][y_wine == label],
               X_wine[:,2][y_wine == label],
               marker=marker,
               color=color,
               s=40,
               alpha=0.7,
               label='class {}'.format(label))

ax.set_xlabel('alcohol by volume in percent')
ax.set_ylabel('malic acid in g/l')
ax.set_zlabel('ash content in g/l')
plt.legend(loc='upper right')
plt.title('Wine dataset')

plt.show()
```

## Splitting into training and test dataset

[[back to top]](#Sections)

It is a typical procedure for machine learning and pattern classification tasks to split one dataset into two: a training dataset and a test dataset.
The training dataset is henceforth used to train our algorithms or classifier, and the test dataset is a way to validate the outcome quite objectively before we apply it to "new, real world data".

Here, we will split the dataset randomly so that 70% of the total dataset will become our training dataset, and 30% will become our test dataset, respectively.

```python
from sklearn.cross_validation import train_test_split
from sklearn import preprocessing

X_train, X_test, y_train, y_test = train_test_split(X_wine, y_wine,
     test_size=0.30, random_state=123)
```

    /srv/venv/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)

Note that since this a random assignment, the original relative frequencies for each class label are not maintained.

```python
print('Class label frequencies')

print('\nTraining Dataset:')
for l in range(1,4):
    print('Class {:} samples: {:.2%}'.format(l, list(y_train).count(l)/y_train.shape[0]))

print('\nTest Dataset:')
for l in range(1,4):
    print('Class {:} samples: {:.2%}'.format(l, list(y_test).count(l)/y_test.shape[0]))
```

    Class label frequencies

    Training Dataset:
    Class 1 samples: 36.29%
    Class 2 samples: 42.74%
    Class 3 samples: 20.97%

    Test Dataset:
    Class 1 samples: 25.93%
    Class 2 samples: 33.33%
    Class 3 samples: 40.74%

## Feature Scaling

[[back to top]](#Sections)

Another popular procedure is to standardize the data prior to fitting the model and other analyses so that the features will have the properties of a standard normal distribution with

$\mu = 0$ and $\sigma = 1$

where $\mu$ is the mean (average) and $\sigma$ is the standard deviation from the mean, so that the standard scores of the samples are calculated as follows:

\begin{equation} z = \frac{x - \mu}{\sigma}\end{equation}

```python
std_scale = preprocessing.StandardScaler().fit(X_train)
X_train = std_scale.transform(X_train)
X_test = std_scale.transform(X_test)
```

```python
f, ax = plt.subplots(1, 2, sharex=True, sharey=True, figsize=(10,5))

for a,x_dat, y_lab in zip(ax, (X_train, X_test), (y_train, y_test)):

    for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue','red','green')):

        a.scatter(x=x_dat[:,0][y_lab == label],
                y=x_dat[:,1][y_lab == label],
                marker=marker,
                color=color,
                alpha=0.7,
                label='class {}'.format(label)
                )

    a.legend(loc='upper right')

ax[0].set_title('Training Dataset')
ax[1].set_title('Test Dataset')
f.text(0.5, 0.04, 'malic acid (standardized)', ha='center', va='center')
f.text(0.08, 0.5, 'alcohol (standardized)', ha='center', va='center', rotation='vertical')

plt.show()
```

## Linear Transformation & Classification: Multiple Discriminant Analysis (MDA)

[[back to top]](#Sections)

The main purposes of a Multiple Discriminant Analysis is to analyze the data to identify patterns to project it onto a subspace that yields a better separation of the classes. Also, the dimensionality of the dataset shall be reduced with minimal loss of information.

**The approach is very similar to a Principal Component Analysis (PCA), but in addition to finding the component axes that maximize the variance of our data, we are additionally interested in the axes that maximize the separation of our classes (e.g., in a supervised pattern classification problem)**

Here, our desired outcome of the multiple discriminant analysis is to project a feature space (our dataset consisting of n d-dimensional samples) onto a smaller subspace that represents our data "well" and has a good class separation. A possible application would be a pattern classification task, where we want to reduce the computational costs and the error of parameter estimation by reducing the number of dimensions of our feature space by extracting a subspace that describes our data "best".

#### Principal Component Analysis (PCA) Vs. Multiple Discriminant Analysis (MDA)

Both Multiple Discriminant Analysis (MDA) and Principal Component Analysis (PCA) are linear transformation methods and closely related to each other. In PCA, we are interested to find the directions (components) that maximize the variance in our dataset, where in MDA, we are additionally interested to find the directions that maximize the separation (or discrimination) between different classes (for example, in pattern classification problems where our dataset consists of multiple classes. In contrast two PCA, which ignores the class labels).

**In other words, via PCA, we are projecting the entire set of data (without class labels) onto a different subspace, and in MDA, we are trying to determine a suitable subspace to distinguish between patterns that belong to different classes. Or, roughly speaking in PCA we are trying to find the axes with maximum variances where the data is most spread (within a class, since PCA treats the whole data set as one class), and in MDA we are additionally maximizing the spread between classes.**

In typical pattern recognition problems, a PCA is often followed by an MDA.

If you are interested, you can find more information about the MDA in my IPython notebook
[Stepping through a Multiple Discriminant Analysis - using Python's NumPy and matplotlib](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/projection/linear_discriminant_analysis.ipynb?create=1).

Like we did in the PCA section above, we will use a `scikit-learn` funcion, [`sklearn.lda.LDA`](http://scikit-learn.org/stable/modules/generated/sklearn.lda.LDA.html) in order to transform our training data onto 2 dimensional subspace, where MDA is basically the more generalized form of an LDA (Linear Discriminant Analysis):

```python
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
sklearn_lda = LDA(n_components=2)
sklearn_transf = sklearn_lda.fit_transform(X_train, y_train)

plt.figure(figsize=(10,8))

for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue', 'red', 'green')):

    plt.scatter(x=sklearn_transf[:,0][y_train == label],
                y=sklearn_transf[:,1][y_train == label],
                marker=marker,
                color=color,
                alpha=0.7,
                label='class {}'.format(label)
                )

plt.xlabel('vector 1')
plt.ylabel('vector 2')

plt.legend()
plt.title('Most significant singular vectors after linear transformation via LDA')

plt.show()
```

## Classification via LDA

[[back to top]](#Sections)

The LDA that we've just used in the section above can also be used as a simple linear classifier.

```python
# fit model
lda_clf = LDA()
lda_clf.fit(X_train, y_train)
LDA(n_components=None, priors=None)

# prediction
print('1st sample from test dataset classified as:', lda_clf.predict(X_test[0,:].reshape(1, -1)))
print('actual class label:', y_test[0])
```

    1st sample from test dataset classified as: [3]
    actual class label: 3

Another handy subpackage of sklearn is `metrics`. The [`metrics.accuracy_score`](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html), for example, is quite useful to evaluate how many samples can be classified correctly:

```python
from sklearn import metrics
pred_train = lda_clf.predict(X_train)

print('Prediction accuracy for the training dataset')
print('{:.2%}'.format(metrics.accuracy_score(y_train, pred_train)))
```

    Prediction accuracy for the training dataset
    100.00%

To verify that over model was not overfitted to the training dataset, let us evaluate the classifier's accuracy on the test dataset:

```python
pred_test = lda_clf.predict(X_test)

print('Prediction accuracy for the test dataset')
print('{:.2%}'.format(metrics.accuracy_score(y_test, pred_test)))
```

    Prediction accuracy for the test dataset
    98.15%

**Confusion Matrix**
As we can see above, there was a very low misclassification rate when we'd apply the classifier on the test data set. A confusion matrix can tell us in more detail which particular classes could not classified correctly.

	<colgroup width="60"></colgroup>
	<colgroup span="4" width="82"></colgroup>

		predicted class

		class 1
		class 2
		class 3

		actual class
		class 1
		True positives

		class 2

		True positives

		class 3

		True positives

```python
print('Confusion Matrix of the LDA-classifier')
print(metrics.confusion_matrix(y_test, lda_clf.predict(X_test)))
```

    Confusion Matrix of the LDA-classifier
    [[14  0  0]
     [ 1 17  0]
     [ 0  0 22]]

As we can see, one sample from class 2 was incorrectly labeled as class 1, from the perspective of class 1, this would be 1 "False Negative" or a "False Postive" from the perspective of class 2, respectively

## Stochastic Gradient Descent (SGD) as linear classifier

[[back to top]](#Sections)

Let us now compare the classification accuracy of the LDA classifier with a simple classification (we also use the probably not ideal default settings here) via stochastic gradient descent, an algorithm that minimizes a linear objective function.
More information about the `sklearn.linear_model.SGDClassifier` can be found [here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html).

```python
from sklearn.linear_model import SGDClassifier

sgd_clf = SGDClassifier()
sgd_clf.fit(X_train, y_train)

pred_train = sgd_clf.predict(X_train)
pred_test = sgd_clf.predict(X_test)

print('\nPrediction accuracy for the training dataset')
print('{:.2%}\n'.format(metrics.accuracy_score(y_train, pred_train)))

print('Prediction accuracy for the test dataset')
print('{:.2%}\n'.format(metrics.accuracy_score(y_test, pred_test)))

print('Confusion Matrix of the SGD-classifier')
print(metrics.confusion_matrix(y_test, sgd_clf.predict(X_test)))
```

    Prediction accuracy for the training dataset
    99.19%

    Prediction accuracy for the test dataset
    100.00%

    Confusion Matrix of the SGD-classifier
    [[14  0  0]
     [ 0 18  0]
     [ 0  0 22]]

    /srv/venv/lib/python3.6/site-packages/sklearn/linear_model/stochastic_gradient.py:84: FutureWarning: max_iter and tol parameters have been added in <class 'sklearn.linear_model.stochastic_gradient.SGDClassifier'> in 0.19. If both are left unset, they default to max_iter=5 and tol=None. If tol is not None, max_iter defaults to max_iter=1000. From 0.21, default max_iter will be 1000, and default tol will be 1e-3.
      "and default tol will be 1e-3." % type(self), FutureWarning)

```python

```

```python

```

```python

```

```python
test complete; Gopal
```


# An introduction to copulas

## Caveat

To start off, I'd like to caveat that I'll try and do my best to explain what a copula is. I'll do my best to keep it as simple as possible so that a person with some basic concepts of probability and distribution functions can understand it. In doing so, I may be imprecise and quite likely, flat out wrong. To my astute readers, if you do catch a mistake or know a better way of describing the content, please contact me so we can discuss and improve the quality of this article.

### The audience

This is an article for beginners. If you're an undergrad who took some basic courses in math and statistics, then you're in the right place. If you need a primer on some probability and distributions, read this [quick primer](https://copulae.readthedocs.io/en/latest/explainers/quick_primer.html).

There would be *almost* no math here. So no Sklar's theorem, Fretchet-Hoffding bounds, etc, etc. Alrighty, let's start *yet another* introduction to copulas.

## A simple explanation of Copulas

From [Wikipedia](https://en.wikipedia.org/wiki/Copula_(probability_theory))'s definition:

> A copula is a multivariate cumulative distribution function for which the marginal probability distribution of each
variable is uniform on the interval [0, 1]. Copulas are used to describe the dependence between random variables.

What did those 2 sentences mean? This could be a little hard to digest at the start. No worries, we'll run through some simple examples to elucidate the concept. To do so, we'll tackle these few keywords

1. **multivariate cumulative distribution function**
2. **marginal probability distribution** (for which each variable is uniform between the interval [0, 1])

## A simple explanation of Copulas

From [Wikipedia](https://en.wikipedia.org/wiki/Copula_(probability_theory))'s definition:

> A copula is a multivariate cumulative distribution function for which the marginal probability distribution of each
variable is uniform on the interval [0, 1]. Copulas are used to describe the dependence between random variables.

What did those 2 sentences mean? This could be a little hard to digest at the start. No worries, we'll run through some simple examples to elucidate the concept. To do so, we'll tackle these few keywords

1. **multivariate cumulative distribution function**
2. **marginal probability distribution** (for which each variable is uniform between the interval [0, 1])

### Multivariate distributions

In school, we likely have studied multivariate distributions in the form of multivariate Gaussian distributions. This means we have at least 2 data series, each with their own mean and variance, and with their joint distribution described by a correlation matrix.

```python
from scipy.stats import multivariate_normal as mvn

import pandas as pd
import seaborn as sns

# the multivariate gaussian can simply be described by the means and covariance matrix too
cov_matrix = [[2, -0.45],
              [-0.45, 3]]
means = [3.3, -2.1]

mvn_dist = mvn(mean = means, cov = cov_matrix)
mvn_rvs = pd.DataFrame(mvn_dist.rvs(10000), columns=["Margin 1", "Margin 2"])

sns.jointplot("Margin 1", "Margin 2", data=mvn_rvs, kind='kde', height=6)
```

Based on the following parameterizations

```python
cov_matrix = [[2, -0.45],
              [-0.45, 3]]
means = [3.3, -2.1]

# The correlation matrix is
corr_matrix = [[ 1.        , -0.18371173],
               [-0.18371173,  1.        ]]

# There is a helper function to retrieve the correlation matrix from the covariance matrix
from copulae.core import cov2corr, corr2cov
```

We see that this multivariate Gaussian has 2 marginal components. One is a $N(3, \sqrt{2})$ and the other is $N(-2.1, \sqrt{3})$. We could now take two values and plug it into the multivariate Gaussian's CDF to find out what the probability of finding that vector of numbers or less. For example,

```python
def find_prob(a: float, b: float):
    print(f"Probability of seeing ({a}, {b}) or less: {round(mvn_dist.cdf([a, b]), 4)}")

find_prob(-10, -2)
find_prob(2, -2)
find_prob(10, 10)
```

    Probability of seeing (-10, -2) or less: 0.0
    Probability of seeing (2, -2) or less: 0.0744
    Probability of seeing (10, 10) or less: 1.0

Let's write some definitions and what we've just seen in some pseudo math.

$$
\begin{align}
X_1 &\sim N(3, \sqrt{2}) \\
X_2 &\sim N(-2.1, \sqrt(3)) \\
F_1(x_1) &= \text{Normal Marginal 1 CDF} \\
         &= u_1 \\
F_2(x_1) &= \text{Normal Marginal 2 CDF} \\
         &= u_2 \\
F^{-1}_1(u_1) &= \text{Quantile Function 1} \\
              &= x_1 \\
F^{-1}_2(u_2) &= \text{Quantile Function 2} \\
              &= x_2 \\
P(X_1, X_2) &= \text{Probability of seeing the two numbers} \\
            &= c \left( F^{-1}_1(u_1), F^{-1}_2(u_2) \right)
\end{align}
$$

where $c(\cdot)$ is a magical (pdf) function that takes in the marginal values and returns the probability of seeing the values. You might want to ask, why did we write it in such a complicated manner? Why not write is simply as $C(x_1, x_2)$? The answer to that is because this is the *secret* to what a copula is.

In the multivariate Gaussian case, $C$ is the Guassian distribution also defined by the correlation. However, imagine if we could now

1. Switch the function $C$, such that we are no longer restricted to a Gaussian but could also use a Student-T or any other type of dependency structure
2. We could switch out the marginals, such that we could have an Student-T marginal for one and a Laplace distribution for another

As you see in the example above, using a CDF function, we can convert a variable to a number between `[0, 1]`. And using a quantile function, we can convert a variable between `[0, 1]` to any value within the distribution. The flexibility to switch the marginals and dependency structure is in effect what a copula is and what makes it so flexible in fitting data.

### Back to the definitions

> A copula is a multivariate cumulative distribution function for which the marginal probability distribution of each
variable is uniform on the interval `[0, 1]`. Copulas are used to describe the dependence between random variables.

In textbooks, a copula is usually defined like

$$
C(u_1, u_2, \dots, u_n) = P(U_1 \leq u_1, U_2 \leq u_2, \dots, U_n \leq u_n)
$$

Where $U$ symbolizes a uniform random variable between `[0, 1]`. Based on what we've learn above, we start by rewriting parts of the formula.

First off, some pseudo math rewriting

$$
U_1 \leq u_1 \sim \text{looks like a CDF} \\
F_1(x_1) = P(X_1 \leq x_1)
$$

This means, as long as we can convert something to a uniform variable, we can put it in a copula and create some joint distributions. To solidify this concept, let's start by generating some random variables using the concepts we've seen.

### Generating data - "Copula" style

Let's assume we are some super-powered divine entity. Maybe the God of Fortune if you're Taoist. Our daily job is to determine the returns of stocks for those silly human gamblers who call themselves investors. To do our job, we select a marginal distribution for each stock and also a joint distribution for all of them. The joint distribution we call the **dependency structure**.

In this world that we are in, there are only 2 stocks, **A** and **B**. **A** follows a [Student distribution](https://en.wikipedia.org/wiki/Student%27s_t-distribution) with 16 df, 0.05 mean and 1.2 standard deviation. **B** follows a [Laplace distribution](https://en.wikipedia.org/wiki/Laplace_distribution) with a mean of 0.1 and 1.5 standard deviation. The dependency structure is a Gaussian copula with mean `[0, 0]` and covariance `[[1, 0.2], [0.2, 1]]`.

Assuming today is day 0, let's do our job and randomly generate the returns profile for the 2 stocks for the next 10 days.

The pseudo-algorithm for the process is as follows:

```python
import numpy as np
from scipy.stats import laplace, multivariate_normal as mvn, t

np.random.seed(1234)
m1 = t(df=16, loc=0.05, scale=1.2)  # marginal for stock A
m2 = laplace(loc=0.1, scale=1.5)  # marginal for stock B
ds = mvn(mean=[0, 0], cov=[[1, 0.2], [0.2, 1]])  # dependency structure

# first we generate some random variables from the dependency structure
rvs = ds.rvs(size=10)
rvs
```

    array([[ 0.38806706, -1.11841127],
           [-0.91203162, -1.30750847],
           [-0.00292548,  1.11925674],
           [-0.26326151, -1.06840713],
           [ 1.40624015, -1.43055686],
           [-1.51817559, -0.26345209],
           [ 0.5399121 , -2.01679549],
           [ 0.25743544,  0.26011499],
           [-0.49690066, -0.13122506],
           [-0.04501576, -2.00171371]])

Now that we have this set of random variables that describes the dependency structure for the 2 distributions, what we need to do next is to convert these variables into their respective distribution's values. This sounds confusing but is actually really simple.

It's just a bunch of inverse transforms and cdfs.

```python
from scipy.stats import norm

# note that its 0 and 1 because in the multivariate normal distribution, we set the mean and variance
# to 0 and 1 for both the distributions. If we had changed the mean and variance, we just need to alter
# the values accordingly. The scale is standard deviation. So if you used a covariance matrix like
# [[9, 4], [4, 16]], then the scale would be 3 and 4 respectively
x1 = norm(loc=0, scale=1).cdf(rvs[:, 0])
x2 = norm(loc=0, scale=1).cdf(rvs[:, 1])

# remember that CDF converts a value to a number between [0, 1]. Which means the probability of seeing that
# value or less
pd.DataFrame({"x1": x1, "x2": x2}).round(4)
```

      x1
      x2

      0
      0.6510
      0.1317

      1
      0.1809
      0.0955

      2
      0.4988
      0.8685

      3
      0.3962
      0.1427

      4
      0.9202
      0.0763

      5
      0.0645
      0.3961

      6
      0.7054
      0.0219

      7
      0.6016
      0.6026

      8
      0.3096
      0.4478

      9
      0.4820
      0.0227

After getting this variables between `[0, 1]`, you can then apply the inverse transform to get the distribution's values.

```python
t_rvs = t(df=16, loc=0.05, scale=1.2).ppf(x1)
laplace_rvs = laplace(loc=0.1, scale=1.5).ppf(x2)

# Returns of Stock A and B in percentage
pd.DataFrame({"A": t_rvs, "B": laplace_rvs}).round(4)
```

      A
      B

      0
      0.5242
      -1.9012

      1
      -1.0767
      -2.3829

      2
      0.0464
      2.1032

      3
      -0.2712
      -1.7811

      4
      1.8198
      -2.7203

      5
      -1.8710
      -0.2494

      6
      0.7112
      -4.5950

      7
      0.3641
      0.4445

      8
      -0.5581
      -0.0654

      9
      -0.0049
      -4.5412

That's it, that's how you generate random variables.

Now let's explore how different this is from a multivariate distribution. However, let's switch to more extreme distributions to highlight the differences.

```python
from scipy.stats import pareto, weibull_min

def plot_extreme_jointplots(rvs):
    x1 = norm(loc=0, scale=1).cdf(rvs[:, 0])
    x2 = norm(loc=0, scale=1).cdf(rvs[:, 1])
    df1 = pd.DataFrame({
        "A": weibull_min(c=0.8, loc=0.05, scale=1.2).ppf(x1),
        "B": pareto(b=4, loc=0.1, scale=1.5).ppf(x2)
    })

    sns.jointplot("A", "B", data=df1, kind='kde', height=8,
                  xlim=(df1.A.min(), 3), ylim=(df1.B.min(), 3))

def plot_multivariate_normal_jointplots(rvs):
    df1 = pd.DataFrame({
        "A": rvs[:, 0],
        "B": rvs[:, 1]
    })

    sns.jointplot("A", "B", data=df1, kind='kde', height=8)
```

```python
np.random.seed(888)
rvs = ds.rvs(size=10000)

plot_extreme_jointplots(rvs)
```

```python
plot_multivariate_normal_jointplots(rvs)
```

Observe from the two plots above, how we started from a similar dependency structure but somehow got very different
joint distributions. This is essentially a copula.

## The End

That's it for this short introduction. Hope it shed some light on how a copula works. It's a pretty high level overview and ignores a lot of math. I hope to be able to write a more precise document on it in the near future (barring any work commitments 😩).

But more than that, I also hope to write some example use-cases. I find that running through (pseudo) real-world practical examples is usually a better pedagogical tool after all. Till then.