```python
from pomegranate import *
import seaborn
%pylab inline
seaborn.set_style('whitegrid')
numpy.set_printoptions(suppress=True)
```

    Populating the interactive namespace from numpy and matplotlib

    /home/jmschr/anaconda2/lib/python2.7/site-packages/IPython/core/magics/pylab.py:161: UserWarning: pylab import has clobbered these variables: ['random', 'log']
    `%matplotlib` prevents importing * from pylab and numpy
      "\n`%matplotlib` prevents importing * from pylab and numpy"

# Naive Bayes and Bayes Classifiers: A Tutorial

author: Jacob Schreiber

contact: jmschreiber91@gmail.com

Bayes classifiers are some of the simplest machine learning models that exist, due to their intuitive probabilistic interpretation and simple fitting step. Each class is modeled as a probability distribution, and the data is interpreted as samples drawn from these underlying distributions. Fitting the model to data is as simple as calculating maximum likelihood parameters for the data that falls under each class, and making predictions is as simple as using Bayes' rule to determine which class is most likely given the distributions. Bayes' Rule is the following:

\begin{equation}
P(M|D) = \frac{P(D|M)P(M)}{P(D)}
\end{equation}

where M stands for the model and D stands for the data. $P(M)$ is known as the prior, because it is the probability that a sample is of a certain class before you even know what the sample is. This is generally just the frequency of each class. Intuitively, it makes sense that you would want to model this, because if one class occurs 10x more than another class, it is more likely that a given sample will belong to that distribution. $P(D|M)$ is the likelihood, or the probability, of the data under a given model. Lastly, $P(M|D)$ is the posterior, which is the probability of each component of the model, or class, being the component which generated the data. It is called the posterior because the prior corresponds to probabilities before seeing data, and the posterior corresponds to probabilities after observing the data. In cases where the prior is uniform, the posterior is just equal to the normalized likelihoods. This equation forms the basis of most probabilistic modeling, with interesting priors allowing the user to inject sophisticated expert knowledge into the problem directly.

Let's take a look at some single dimensional data in order to introduce these concepts more thoroughly.

```python
X = numpy.concatenate((numpy.random.normal(3, 1, 200), numpy.random.normal(10, 2, 1000)))
y = numpy.concatenate((numpy.zeros(200), numpy.ones(1000)))

x1 = X[:200]
x2 = X[200:]

plt.figure(figsize=(16, 5))
plt.hist(x1, bins=25, color='m', edgecolor='m', label="Class A")
plt.hist(x2, bins=25, color='c', edgecolor='c', label="Class B")
plt.xlabel("Value", fontsize=14)
plt.ylabel("Count", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

The data seems like it comes from two normal distributions, with the cyan class being more prevalent than the magenta class. A natural way to model this data would be to create a normal distribution for the cyan data, and another for the magenta distribution.

Let's take a look at doing that. All we need to do is use the `from_samples` class method of the `NormalDistribution` class.

```python
d1 = NormalDistribution.from_samples(x1)
d2 = NormalDistribution.from_samples(x2)
idxs = numpy.arange(0, 15, 0.1)

p1 = map(d1.probability, idxs)
p2 = map(d2.probability, idxs)

plt.figure(figsize=(16, 5))
plt.plot(idxs, p1, color='m'); plt.fill_between(idxs, 0, p1, facecolor='m', alpha=0.2)
plt.plot(idxs, p2, color='c'); plt.fill_between(idxs, 0, p2, facecolor='c', alpha=0.2)
plt.xlabel("Value", fontsize=14)
plt.ylabel("Probability", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

It looks like some aspects of the data are captured well by doing things this way-- specifically the mean and variance of the normal distributions. This allows us to easily calculate $P(D|M)$ as the probability of a sample under either the cyan or magenta distributions using the normal (or Gaussian) probability density equation:

\begin{align}
P(D|M) &= P(x|\mu, \sigma) \\
       &= \frac{1}{\sqrt{2\pi\sigma^{2}}} exp \left(-\frac{(x-u)^{2}}{2\sigma^{2}} \right)
\end{align}

However, if we look at the original data, we see that the cyan distributions is both much wider than the purple distribution and much taller, as there were more samples from that class in general. If we reduce that data down to these two distributions, we lose the class imbalance. We want our prior to model this class imbalance, with the reasoning being that if we randomly draw a sample from the samples observed thus far, it is far more likely to be a cyan than a magenta sample. Let's take a look at this class imbalance exactly.

```python
magenta_prior = 1. * len(x1) / len(X)
cyan_prior = 1. * len(x2) / len(X)

plt.figure(figsize=(4, 6))
plt.title("Prior Probabilities P(M)", fontsize=14)
plt.bar(0, magenta_prior, facecolor='m', edgecolor='m')
plt.bar(1, cyan_prior, facecolor='c', edgecolor='c')
plt.xticks([0, 1], ['P(Magenta)', 'P(Cyan)'], fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

The prior $P(M)$ is a vector of probabilities over the classes that the model can predict, also known as components. In this case, if we draw a sample randomly from the data that we have, there is a ~83% chance that it will come from the cyan class and a ~17% chance that it will come from the magenta class.

Let's multiply the probability densities we got before by this imbalance.

```python
d1 = NormalDistribution.from_samples(x1)
d2 = NormalDistribution.from_samples(x2)
idxs = numpy.arange(0, 15, 0.1)

p_magenta = numpy.array(map(d1.probability, idxs)) * magenta_prior
p_cyan = numpy.array(map(d2.probability, idxs)) * cyan_prior

plt.figure(figsize=(16, 5))
plt.plot(idxs, p_magenta, color='m'); plt.fill_between(idxs, 0, p_magenta, facecolor='m', alpha=0.2)
plt.plot(idxs, p_cyan, color='c'); plt.fill_between(idxs, 0, p_cyan, facecolor='c', alpha=0.2)
plt.xlabel("Value", fontsize=14)
plt.ylabel("P(M)P(D|M)", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

This looks a lot more faithful to the original data, and actually corresponds to $P(M)P(D|M)$, the prior multiplied by the likelihood. However, these aren't actually probability distributions anymore, as they no longer integrate to 1. This is why the $P(M)P(D|M)$ term has to be normalized by the $P(D)$ term in Bayes' rule in order to get a probability distribution over the components. However, $P(D)$ is difficult to determine exactly-- what is the probability of the data? Well, we can sum over the classes to get that value, since $P(D) = \sum_{i=1}^{c} P(D|M)P(M)$ for a problem with c classes. This translates into $P(D) = P(M=Cyan)P(D|M=Cyan) + P(M=Magenta)P(D|M=Magenta)$ for this specific problem, and those values can just be pulled from the unnormalized plots above.

This gives us the full Bayes' rule, with the posterior $P(M|D)$ being the proportion of density of the above plot coming from each of the two distributions at any point on the line. Let's take a look at the posterior probabilities of the two classes on the same line.

```python
magenta_posterior = p_magenta / (p_magenta + p_cyan)
cyan_posterior = p_cyan / (p_magenta + p_cyan)

plt.figure(figsize=(16, 5))
plt.subplot(211)
plt.plot(idxs, p_magenta, color='m'); plt.fill_between(idxs, 0, p_magenta, facecolor='m', alpha=0.2)
plt.plot(idxs, p_cyan, color='c'); plt.fill_between(idxs, 0, p_cyan, facecolor='c', alpha=0.2)
plt.xlabel("Value", fontsize=14)
plt.ylabel("P(M)P(D|M)", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)

plt.subplot(212)
plt.plot(idxs, magenta_posterior, color='m')
plt.plot(idxs, cyan_posterior, color='c')
plt.xlabel("Value", fontsize=14)
plt.ylabel("P(M|D)", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

The top plot shows the same densities as before, while the bottom plot shows the proportion of the density belonging to either class at that point. This proportion is known as the posterior $P(M|D)$, and can be interpreted as the probability of that point belonging to each class. This is one of the native benefits of probabilistic models, that instead of providing a hard class label for each sample, they can provide a soft label in the form of the probability of belonging to each class.

We can implement all of this simply in pomegranate using the `NaiveBayes` class.

```python
idxs = idxs.reshape(idxs.shape[0], 1)
X = X.reshape(X.shape[0], 1)

model = NaiveBayes.from_samples(NormalDistribution, X, y)
posteriors = model.predict_proba(idxs)

plt.figure(figsize=(14, 4))
plt.plot(idxs, posteriors[:,0], color='m')
plt.plot(idxs, posteriors[:,1], color='c')
plt.xlabel("Value", fontsize=14)
plt.ylabel("P(M|D)", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

Looks like we're getting the same plots for the posteriors just through fitting the naive Bayes model directly to data. The predictions made will come directly from the posteriors in this plot, with cyan predictions happening whenever the cyan posterior is greater than the magenta posterior, and vice-versa.

## Naive Bayes

In the univariate setting, naive Bayes is identical to a general Bayes classifier. The divergence occurs in the multivariate setting, the naive Bayes model assumes independence of all features, while a Bayes classifier is more general and can support more complicated interactions or covariances between features. Let's take a look at what this means in terms of Bayes' rule.

\begin{align}
P(M|D) &= \frac{P(M)P(D|M)}{P(D)} \\
       &= \frac{P(M)\prod_{i=1}^{d}P(D_{i}|M_{i})}{P(D)}
\end{align}

This looks fairly simple to compute, as we just need to pass each dimension into the appropriate distribution and then multiply the returned probabilities together. This simplicity is one of the reasons why naive Bayes is so widely used. Let's look closer at using this in pomegranate, starting off by generating two blobs of data that overlap a bit and inspecting them.

```python
X = numpy.concatenate([numpy.random.normal(3, 2, size=(150, 2)), numpy.random.normal(7, 1, size=(250, 2))])
y = numpy.concatenate([numpy.zeros(150), numpy.ones(250)])

plt.figure(figsize=(8, 8))
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c')
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m')
plt.xlim(-2, 10)
plt.ylim(-4, 12)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

Now, let's fit our naive Bayes model to this data using pomegranate. We can use the `from_samples` class method, pass in the distribution that we want to model each dimension, and then the data. We choose to use `NormalDistribution` in this particular case, but any supported distribution would work equally well, such as `BernoulliDistribution` or `ExponentialDistribution`. To ensure we get the correct decision boundary, let's also plot the boundary recovered by sklearn.

```python
from sklearn.naive_bayes import GaussianNB

model = NaiveBayes.from_samples(NormalDistribution, X, y)
clf = GaussianNB().fit(X, y)

xx, yy = np.meshgrid(np.arange(-2, 10, 0.02), np.arange(-4, 12, 0.02))
Z1 = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
Z2 = clf.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

plt.figure(figsize=(16, 8))
plt.subplot(121)
plt.title("pomegranate naive Bayes", fontsize=16)
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c')
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m')
plt.contour(xx, yy, Z1)
plt.xlim(-2, 10)
plt.ylim(-4, 12)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)

plt.subplot(122)
plt.title("sklearn naive Bayes", fontsize=16)
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c')
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m')
plt.contour(xx, yy, Z2)
plt.xlim(-2, 10)
plt.ylim(-4, 12)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

Drawing the decision boundary helps to verify that we've produced a good result by cleanly splitting the two blobs from each other.

Bayes' rule provides a great deal of flexibility in terms of what the actually likelihood functions are. For example, when considering a multivariate distribution, there is no need for each dimension to be modeled by the same distribution. In fact, each dimension can be modeled by a different distribution, as long as we can multiply the $P(D|M)$ terms together.

Let's consider the example of some noisy signals that have been segmented. We know that they come from two underlying phenomena, the cyan phenomena and the magenta phenomena, and want to classify future segments. To do this, we have three features-- the mean signal of the segment, the standard deviation, and the duration.

```python
def plot_signal(X, n):
    plt.figure(figsize=(16, 6))
    t_current = 0
    for i in range(n):
        mu, std, t = X[i]
        chunk = numpy.random.normal(mu, std, t)
        plt.plot(numpy.arange(t_current, t_current+t), chunk, c='cm'[i % 2])
        t_current += t

    plt.xticks(fontsize=14)
    plt.yticks(fontsize=14)
    plt.xlabel("Time (s)", fontsize=14)
    plt.ylabel("Signal", fontsize=14)
    plt.ylim(20, 40)
    plt.show()

def create_signal(n):
    X, y = [], []
    for i in range(n):
        mu = numpy.random.normal(30.0, 0.4)
        std = numpy.random.lognormal(-0.1, 0.4)
        t = int(numpy.random.exponential(50)) + 1
        X.append([mu, std, t])
        y.append(0)

        mu = numpy.random.normal(30.5, 0.8)
        std = numpy.random.lognormal(-0.3, 0.6)
        t = int(numpy.random.exponential(200)) + 1
        X.append([mu, std, t])
        y.append(1)

    return numpy.array(X), numpy.array(y)

X_train, y_train = create_signal(1000)
X_test, y_test = create_signal(250)
plot_signal(X_train, 20)
```

We can start by modeling each variable as Gaussians, like before, and see what accuracy we get.

```python
model = NaiveBayes.from_samples(NormalDistribution, X_train, y_train)
print "Gaussian Naive Bayes: ", (model.predict(X_test) == y_test).mean()

clf = GaussianNB().fit(X_train, y_train)
print "sklearn Gaussian Naive Bayes: ", (clf.predict(X_test) == y_test).mean()
```

    Gaussian Naive Bayes:  0.798
    sklearn Gaussian Naive Bayes:  0.798

We get identical values for sklearn and for pomegranate, which is good. However, let's take a look at the data itself to see whether a Gaussian distribution is the appropriate distribution for the data.

```python
plt.figure(figsize=(14, 4))
plt.subplot(131)
plt.title("Mean")
plt.hist(X_train[y_train == 0, 0], color='c', alpha=0.5, bins=25)
plt.hist(X_train[y_train == 1, 0], color='m', alpha=0.5, bins=25)

plt.subplot(132)
plt.title("Standard Deviation")
plt.hist(X_train[y_train == 0, 1], color='c', alpha=0.5, bins=25)
plt.hist(X_train[y_train == 1, 1], color='m', alpha=0.5, bins=25)

plt.subplot(133)
plt.title("Duration")
plt.hist(X_train[y_train == 0, 2], color='c', alpha=0.5, bins=25)
plt.hist(X_train[y_train == 1, 2], color='m', alpha=0.5, bins=25)
plt.show()
```

So, unsurprisingly (since you can see that I used non-Gaussian distributions to generate the data originally), it looks like only the mean follows a normal distribution, whereas the standard deviation seems to follow either a gamma or a log-normal distribution. We can take advantage of that by explicitly using these distributions instead of approximating them as normal distributions. pomegranate is flexible enough to allow for this, whereas sklearn currently is not.

```python
model = NaiveBayes.from_samples(NormalDistribution, X_train, y_train)
print "Gaussian Naive Bayes: ", (model.predict(X_test) == y_test).mean()

clf = GaussianNB().fit(X_train, y_train)
print "sklearn Gaussian Naive Bayes: ", (clf.predict(X_test) == y_test).mean()

model = NaiveBayes.from_samples([NormalDistribution, LogNormalDistribution, ExponentialDistribution], X_train, y_train)
print "Heterogeneous Naive Bayes: ", (model.predict(X_test) == y_test).mean()
```

    Gaussian Naive Bayes:  0.798
    sklearn Gaussian Naive Bayes:  0.798
    Heterogeneous Naive Bayes:  0.844

It looks like we're able to get a small improvement in accuracy just by using appropriate distributions for the features, without any type of data transformation or filtering. This certainly seems worthwhile if you can determine what the appropriate underlying distribution is.

Next, there's obviously the issue of speed. Let's compare the speed of the pomegranate implementation and the sklearn implementation.

```python
%timeit GaussianNB().fit(X_train, y_train)
%timeit NaiveBayes.from_samples(NormalDistribution, X_train, y_train)
%timeit NaiveBayes.from_samples([NormalDistribution, LogNormalDistribution, ExponentialDistribution], X_train, y_train)
```

    100 loops, best of 3: 2.45 ms per loop
    1000 loops, best of 3: 2.03 ms per loop
    100 loops, best of 3: 1.99 ms per loop

Looks as if on this small dataset they're all taking approximately the same time. This is pretty much expected, as the fitting step is fairly simple and both implementations use C-level numerics for the calculations. We can give a more thorough treatment of the speed comparison on larger datasets. Let's look at the average time it takes to fit a model to data of increasing dimensionality across 25 runs.

```python
pom_time, skl_time = [], []

n1, n2 = 15000, 60000,
for d in range(1, 101, 5):
    X = numpy.concatenate([numpy.random.normal(3, 2, size=(n1, d)), numpy.random.normal(7, 1, size=(n2, d))])
    y = numpy.concatenate([numpy.zeros(n1), numpy.ones(n2)])

    tic = time.time()
    for i in range(25):
        GaussianNB().fit(X, y)
    skl_time.append((time.time() - tic) / 25)

    tic = time.time()
    for i in range(25):
        NaiveBayes.from_samples(NormalDistribution, X, y)
    pom_time.append((time.time() - tic) / 25)
```

```python
plt.figure(figsize=(14, 6))
plt.plot(range(1, 101, 5), pom_time, color='c', label="pomegranate")
plt.plot(range(1, 101, 5), skl_time, color='m', label="sklearn")
plt.xticks(fontsize=14)
plt.xlabel("Number of Dimensions", fontsize=14)
plt.yticks(fontsize=14)
plt.ylabel("Time (s)")
plt.legend(fontsize=14)
plt.show()
```

It appears as if the two implementations are basically the same speed. This is unsurprising given the simplicity of the calculations, and as mentioned before, the low level implementation.

## Bayes Classifiers

The natural generalization of the naive Bayes classifier is to allow any multivariate function take the place of $P(D|M)$ instead of it being the product of several univariate probability distributions. One immediate difference is that now instead of creating a Gaussian model with effectively a diagonal covariance matrix, you can now create one with a full covariance matrix. Let's see an example of that at work.

```python
tilt_a = [[-2, 0.5], [5, 2]]
tilt_b = [[-1, 1.5], [3, 3]]

X = numpy.concatenate((numpy.random.normal(4, 1, size=(250, 2)).dot(tilt_a), numpy.random.normal(3, 1, size=(800, 2)).dot(tilt_b)))
y = numpy.concatenate((numpy.zeros(250), numpy.ones(800)))

model_a = NaiveBayes.from_samples(NormalDistribution, X, y)
model_b = BayesClassifier.from_samples(MultivariateGaussianDistribution, X, y)

xx, yy = np.meshgrid(np.arange(-5, 30, 0.02), np.arange(0, 25, 0.02))
Z1 = model_a.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
Z2 = model_b.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

plt.figure(figsize=(18, 8))
plt.subplot(121)
plt.contour(xx, yy, Z1)
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c', alpha=0.3)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m', alpha=0.3)
plt.xlim(-5, 30)
plt.ylim(0, 25)

plt.subplot(122)
plt.contour(xx, yy, Z2)
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c', alpha=0.3)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m', alpha=0.3)
plt.xlim(-5, 30)
plt.ylim(0, 25)
plt.show()
```

It looks like we are able to get a better boundary between the two blobs of data. The primary for this is because the data don't form spherical clusters, like you assume when you force a diagonal covariance matrix, but are tilted ellipsoids, that can be better modeled by a full covariance matrix. We can quantify this quickly by looking at performance on the training data.

```python
print "naive training accuracy: {:4.4}".format((model_a.predict(X) == y).mean())
print "bayes classifier training accuracy: {:4.4}".format((model_b.predict(X) == y).mean())
```

    naive training accuracy: 0.9286
    bayes classifier training accuracy: 0.9657

Looks like there is a significant boost. Naturally you'd want to evaluate the performance of the model on separate validation data, but for the purposes of demonstrating the effect of a full covariance matrix this should be sufficient.

While using a full covariance matrix is certainly more complicated than using only the diagonal, there is no reason that the $P(D|M)$ has to even be a single simple distribution versus a full probabilistic model. After all, all probabilistic models, including general mixtures, hidden Markov models, and Bayesian networks, can calculate $P(D|M)$. Let's take a look at an example of using a mixture model instead of a single gaussian distribution.

```python
X = numpy.empty(shape=(0, 2))
X = numpy.concatenate((X, numpy.random.normal(4, 1, size=(200, 2)).dot([[-2, 0.5], [2, 0.5]])))
X = numpy.concatenate((X, numpy.random.normal(3, 1, size=(350, 2)).dot([[-1, 2], [1, 0.8]])))
X = numpy.concatenate((X, numpy.random.normal(7, 1, size=(700, 2)).dot([[-0.75, 0.8], [0.9, 1.5]])))
X = numpy.concatenate((X, numpy.random.normal(6, 1, size=(120, 2)).dot([[-1.5, 1.2], [0.6, 1.2]])))
y = numpy.concatenate((numpy.zeros(550), numpy.ones(820)))

model_a = BayesClassifier.from_samples(MultivariateGaussianDistribution, X, y)

gmm_a = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, 2, X[y == 0])
gmm_b = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, 2, X[y == 1])
model_b = BayesClassifier([gmm_a, gmm_b], weights=numpy.array([1-y.mean(), y.mean()]))

xx, yy = np.meshgrid(np.arange(-10, 10, 0.02), np.arange(0, 25, 0.02))
Z1 = model_a.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
Z2 = model_b.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
centroids1 = numpy.array([distribution.mu for distribution in model_a.distributions])
centroids2 = numpy.concatenate([[distribution.mu for distribution in component.distributions] for component in model_b.distributions])

plt.figure(figsize=(18, 8))
plt.subplot(121)
plt.contour(xx, yy, Z1)
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c', alpha=0.3)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m', alpha=0.3)
plt.scatter(centroids1[:,0], centroids1[:,1], color='k', s=100)

plt.subplot(122)
plt.contour(xx, yy, Z2)
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c', alpha=0.3)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m', alpha=0.3)
plt.scatter(centroids2[:,0], centroids2[:,1], color='k', s=100)
plt.show()
```

Frequently in the real world you will end up with data that looks like the ones in the above plots, not neatly falling into any single simple distribution. Using a mixture here allowed us to model the various components separately and get a more sophisticated decision boundary that doesn't seem to be extremely overfit.

If one wanted to use hidden Markov models to model sequences, all that needs to change is passing in `HiddenMarkovModel` objects instead of `GeneralMixtureModel` ones. Likewise, Bayesian networks are similarly supported and can be passed in just as easily. In fact, the most stacking one could reasonable do is to create a Bayes classifier that distinguishes mixtures of hidden Markov models with mixture emissions (GMM-HMM-GMMs) from each other!


# Bayesian Analysis Using PyMC3

Statistical inference is a process of learning from incomplete or imperfect (error-contaminated) data. Can account for this "imperfection" using either a sampling model or a measurement error model.

### Statistical hypothesis testing

The *de facto* standard for statistical inference is statistical hypothesis testing. The goal of hypothesis testing is to evaluate a **null hypothesis**. There are two possible outcomes:

- reject the null hypothesis
- fail to reject the null hypothesis

Rejection occurs when a chosen test statistic is higher than some pre-specified threshold valuel; non-rejection occurs otherwise.

Notice that neither outcome says anything about the quantity of interest, the **research hypothesis**.

Setting up a statistical test involves several subjective choices by the user that are rarely justified based on the problem or decision at hand:

- statistical test to use
- null hypothesis to test
- significance level

Choices are often based on arbitrary criteria, including "statistical tradition" (Johnson 1999). The resulting evidence is indirect, incomplete, and typically overstates the evidence against the null hypothesis (Goodman 1999).

Most importantly to applied users, the results of statistical hypothesis tests are very easy to misinterpret.

### Estimation

Instead of testing, a more informative and effective approach for inference is based on **estimation** (be it frequentist or Bayesian). That is, rather than testing whether two groups are different, we instead pursue an estimate of *how different* they are, which is fundamentally more informative.

Additionally, we include an estimate of **uncertainty** associated with that difference which includes uncertainty due to our lack of knowledge of the model parameters (*epistemic uncertainty*) and uncertainty due to the inherent stochasticity of the system (*aleatory uncertainty*).

## One Group

Before we compare two groups using Bayesian analysis, let's start with an even simpler scenario: statistical inference for one group.

For this we will use Gelman et al.'s (2007) radon dataset. In this dataset the amount of the radioactive gas radon has been measured among different households in all counties of several states. Radon gas is known to be the highest cause of lung cancer in non-smokers. It is believed to be more strongly present in households containing a basement and to differ in amount present among types of soil.

>  the US EPA has set an action level of 4 pCi/L. At or above this level of radon, the EPA recommends you take corrective measures to reduce your exposure to radon gas.

Let's import the dataset:

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns
sns.set_context('notebook')

import warnings
warnings.filterwarnings("ignore")
```

```python
radon = pd.read_csv('../data/radon.csv', index_col=0)
radon.head()
```

      idnum
      state
      state2
      stfips
      zip
      region
      typebldg
      floor
      room
      basement
      ...
      pcterr
      adjwt
      dupflag
      zipflag
      cntyfips
      county
      fips
      Uppm
      county_code
      log_radon

      0
      5081.0
      MN
      MN
      27.0
      55735
      5.0
      1.0
      1.0
      3.0
      N
      ...
      9.7
      1146.499190
      1.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.832909

      1
      5082.0
      MN
      MN
      27.0
      55748
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      14.5
      471.366223
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.832909

      2
      5083.0
      MN
      MN
      27.0
      55748
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      9.6
      433.316718
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      1.098612

      3
      5084.0
      MN
      MN
      27.0
      56469
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      24.3
      461.623670
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.095310

      4
      5085.0
      MN
      MN
      27.0
      55011
      3.0
      1.0
      0.0
      4.0
      Y
      ...
      13.8
      433.316718
      0.0
      0.0
      3.0
      ANOKA
      27003.0
      0.428565
      1
      1.163151

5 rows × 29 columns

Let's focus on the (log) radon levels measured in a single county (Hennepin).

Suppose we are interested in:

- whether the mean log-radon value is greater than 4 pCi/L in Hennepin county
- the probability that any randomly-chosen household in Hennepin county has a reading of greater than 4

```python
hennepin_radon = radon.query('county=="HENNEPIN"').log_radon
sns.distplot(hennepin_radon);
```

### The model

Recall that the first step in Bayesian inference is specifying a **full probability model** for the problem.

This consists of:

- a likelihood function(s) for the observations
- priors for all unknown quantities

The measurements look approximately normal, so let's start by assuming a normal distribution as the sampling distribution (likelihood) for the data.

$$y_i \sim N(\mu, \sigma^2)$$

(don't worry, we can evaluate this assumption)

This implies that we have 2 unknowns in the model; the mean and standard deviation of the distribution.

#### Prior choice

How do we choose distributions to use as priors for these parameters?

There are several considerations:

- discrete vs continuous values
- the support of the variable
- the available prior information

While there may likely be prior information about the distribution of radon values, we will assume no prior knowledge, and specify a **diffuse** prior for each parameter.

Since the mean can take any real value (since it is on the log scale), we will use another normal distribution here, and specify a large variance to allow the possibility of very large or very small values:

$$\mu \sim N(0, 10^2)$$

For the standard deviation, we know that the true value must be positive (no negative variances!). I will choose a uniform prior bounded from below at zero and from above at a value that is sure to be higher than any plausible value the true standard deviation (on the log scale) could take.

$$\sigma \sim U(0, 10)$$

We can encode these in a Python model, using the PyMC3 package, as follows:

```python
from pymc3 import Model, Normal, Uniform

with Model() as radon_model:

    μ = Normal('μ', mu=0, sd=10)
    σ = Uniform('σ', 0, 10)
```

> ## Software
> Today there is an array of software choices for Bayesians in Python and R, including Stan, PyMC, emcee, TensorFlow Probability. These examples can be replicated in any of these environments.

All that remains is to add the likelihood, which takes $\mu$ and $\sigma$ as parameters, and the log-radon values as the set of observations:

```python
with radon_model:

    dist = Normal('dist', mu=μ, sd=σ, observed=hennepin_radon)
```

Now, we will fit the model using a numerical approach called **Markov chain Monte Carlo (MCMC)**. This will draw samples from the posterior distribution (which cannot be calculated exactly).

```python
from pymc3 import sample

with radon_model:

    samples = sample(1000, tune=1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [σ, μ]
    Sampling 2 chains: 100%|██████████| 4000/4000 [00:02<00:00, 1878.11draws/s]

```python
from pymc3 import plot_posterior

plot_posterior(samples, varnames=['μ'], ref_val=np.log(4), color='LightSeaGreen');
```

The plot shows the posterior distribution of $\mu$, along with an estimate of the 95% posterior **credible interval**.

The output

    83% < 1.38629 < 17%

informs us that the probability of $\mu$ being less than log(4) is 83% and the corresponding probability of being greater than log(4) is 17%.

> The posterior probability that the mean level of household radon in Henneprin County is greater than 4 pCi/L is 0.17.

### Prediction

What is the probability that a given household has a log-radon measurement larger than one? To answer this, we make use of the **posterior predictive distribution**.

$$p(z |y) = \int_{\theta} p(z |\theta) p(\theta | y) d\theta$$

where here $z$ is the predicted value and y is the data used to fit the model.

We can estimate this from the posterior samples of the parameters in the model.

```python
mus = samples['μ']
sigmas = samples['σ']
```

```python
radon_samples = Normal.dist(mus, sigmas).random()
```

```python
(radon_samples > np.log(4)).mean()
```

    0.4495

> The posterior probability that a randomly-selected household in Henneprin County contains radon levels in excess of 4 pCi/L is ~0.47.

### Model checking

But, ***how do we know this model is any good?***

Its important to check the fit of the model, to see if its assumptions are reasonable. One way to do this is to perform **posterior predictive checks**. This involves generating simulated data using the model that you built, and comparing that data to the observed data.

One can choose a particular statistic to compare, such as tail probabilities or quartiles, but here it is useful to compare them graphically.

We already have these simulations from the previous exercise!

```python
sns.distplot(radon_samples, label='simulated')
sns.distplot(hennepin_radon, label='observed')
plt.legend();
```

### Prior sensitivity

Its also important to check the sensitivity of your choice of priors to the resulting inference.

Here is the same model, but with drastically different (though still uninformative) priors specified:

```python
from pymc3 import Flat, HalfCauchy

with Model() as prior_sensitivity:

    μ = Flat('μ')
    σ = HalfCauchy('σ', 5)

    dist = Normal('dist', mu=μ, sd=σ, observed=hennepin_radon)

    sensitivity_samples = sample(1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [σ, μ]
    Sampling 2 chains: 100%|██████████| 3000/3000 [00:01<00:00, 2716.97draws/s]

```python
plot_posterior(sensitivity_samples, varnames=['μ'], ref_val=np.log(4), color='LightSeaGreen');
```

Here is the original model for comparison:

```python
plot_posterior(samples, varnames=['μ'], ref_val=np.log(4), color='LightSeaGreen');
```

## Two Groups with Continiuous Outcome

To illustrate how this Bayesian estimation approach works in practice, we will use a fictitious example from Kruschke (2012) concerning the evaluation of a clinical trial for drug evaluation. The trial aims to evaluate the efficacy of a "smart drug" that is supposed to increase intelligence by comparing IQ scores of individuals in a treatment arm (those receiving the drug) to those in a control arm (those recieving a placebo). There are 47 individuals and 42 individuals in the treatment and control arms, respectively.

```python
drug = pd.DataFrame(dict(iq=(101,100,102,104,102,97,105,105,98,101,100,123,105,103,100,95,102,106,
        109,102,82,102,100,102,102,101,102,102,103,103,97,97,103,101,97,104,
        96,103,124,101,101,100,101,101,104,100,101),
                         group='drug'))
placebo = pd.DataFrame(dict(iq=(99,101,100,101,102,100,97,101,104,101,102,102,100,105,88,101,100,
           104,100,100,100,101,102,103,97,101,101,100,101,99,101,100,100,
           101,100,99,101,100,102,99,100,99),
                            group='placebo'))

trial_data = pd.concat([drug, placebo], ignore_index=True)
trial_data.hist('iq', by='group');
```

Since there appear to be extreme ("outlier") values in the data, we will choose a Student-t distribution to describe the distributions of the scores in each group. This sampling distribution adds **robustness** to the analysis, as a T distribution is less sensitive to outlier observations, relative to a normal distribution.

The three-parameter Student-t distribution allows for the specification of a mean $\mu$, a precision (inverse-variance) $\lambda$ and a degrees-of-freedom parameter $\nu$:

$$f(x|\mu,\lambda,\nu) = \frac{\Gamma(\frac{\nu + 1}{2})}{\Gamma(\frac{\nu}{2})} \left(\frac{\lambda}{\pi\nu}\right)^{\frac{1}{2}} \left[1+\frac{\lambda(x-\mu)^2}{\nu}\right]^{-\frac{\nu+1}{2}}$$

the degrees-of-freedom parameter essentially specifies the "normality" of the data, since larger values of $\nu$ make the distribution converge to a normal distribution, while small values (close to zero) result in heavier tails.

Thus, the likelihood functions of our model are specified as follows:

$$\begin{align}
y^{(drug)}_i &\sim T(\nu, \mu_1, \sigma_1) \\
y^{(placebo)}_i &\sim T(\nu, \mu_2, \sigma_2)
\end{align}$$

As a simplifying assumption, we will assume that the degree of normality $\nu$ is the same for both groups.

### Prior choice

Since the means are real-valued, we will apply normal priors. Since we know something about the population distribution of IQ values, we will center the priors at 100, and use a standard deviation that is more than wide enough to account for plausible deviations from this population mean:

$$\mu_k \sim N(100, 10^2)$$

```python
with Model() as drug_model:

    μ_0 = Normal('μ_0', 100, sd=10)
    μ_1 = Normal('μ_1', 100, sd=10)
```

Similarly, we will use a uniform prior for the standard deviations, with an upper bound of 20.

```python
with drug_model:
    σ_0 = Uniform('σ_0', lower=0, upper=20)
    σ_1 = Uniform('σ_1', lower=0, upper=20)
```

For the degrees-of-freedom parameter $\nu$, we will use an **exponential** distribution with a mean of 30; this allocates high prior probability over the regions of the parameter that describe the range from normal to heavy-tailed data under the Student-T distribution.

```python
from pymc3 import Exponential

with drug_model:
    ν = Exponential('ν_minus_one', 1/29.) + 1

```

```python
sns.distplot(Exponential.dist(1/29).random(size=10000), kde=False);
```

```python
from pymc3 import StudentT

with drug_model:

    drug_like = StudentT('drug_like', nu=ν, mu=μ_1, lam=σ_1**-2, observed=drug.iq)
    placebo_like = StudentT('placebo_like', nu=ν, mu=μ_0, lam=σ_0**-2, observed=placebo.iq)
```

Now that the model is fully specified, we can turn our attention to tracking the posterior quantities of interest. Namely, we can calculate the difference in means between the drug and placebo groups.

As a joint measure of the groups, we will also estimate the "effect size", which is the difference in means scaled by the pooled estimates of standard deviation. This quantity can be harder to interpret, since it is no longer in the same units as our data, but it is a function of all four estimated parameters.

```python
from pymc3 import Deterministic

with drug_model:

    diff_of_means = Deterministic('difference of means', μ_1 - μ_0)

    effect_size = Deterministic('effect size',
                        diff_of_means / np.sqrt((σ_1**2 + σ_0**2) / 2))

```

```python
with drug_model:

    drug_trace = sample(1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [ν_minus_one, σ_1, σ_0, μ_1, μ_0]
    Sampling 2 chains: 100%|██████████| 3000/3000 [00:02<00:00, 1045.66draws/s]

```python
plot_posterior(drug_trace,
                varnames=['μ_0', 'μ_1', 'σ_0', 'σ_1', 'ν_minus_one'],
                color='#87ceeb');
```

```python
plot_posterior(drug_trace,
          varnames=['difference of means', 'effect size'],
          ref_val=0,
          color='#87ceeb');
```

> The posterior probability that the mean IQ of subjects in the treatment group is greater than that of the control group is 0.99.

## Two Groups with Binary Outcome

Now that we have seen how to generalize normally-distributed data to another distribution, we are equipped to analyze other data types. Binary outcomes are common in clinical research:

- survival/death
- true/false
- presence/absence
- positive/negative

In practice, binary outcomes are encoded as ones (for event occurrences) and zeros (for non-occurrence). A single binary variable is distributed as a **Bernoulli** random variable:

$$f(x \mid p) = p^{x} (1-p)^{1-x}$$

Such events are sometimes reported as sums of individual events, such as the number of individuals in a group who test positive for a condition of interest. Sums of Bernoulli events are distributed as **binomial** random variables.

$$f(x \mid n, p) = \binom{n}{x} p^x (1-p)^{n-x}$$

The parameter in both models is $p$, the probability of the occurrence of an event. In terms of inference, we are typically interested in whether $p$ is larger or smaller in one group relative to another.

To demonstrate the comparison of two groups with binary outcomes using Bayesian inference, we will use a sample pediatric dataset. Data on 671 infants with very low (<1600 grams) birth weight from 1981-87 were collected at Duke University Medical Center. Of interest is the relationship between the outcome intra-ventricular hemorrhage (IVH) and predictor such as birth weight, gestational age, presence of pneumothorax and mode of delivery.

```python
vlbw = pd.read_csv('../data/vlbw.csv', index_col=0).dropna(axis=0, subset=['ivh', 'pneumo'])
vlbw.head()
```

      birth
      exit
      hospstay
      lowph
      pltct
      race
      bwt
      gest
      inout
      twn
      ...
      vent
      pneumo
      pda
      cld
      pvh
      ivh
      ipe
      year
      sex
      dead

      5
      81.593002
      81.598999
      2.0
      6.969997
      54.0
      black
      925.0
      28.0
      born at Duke
      0.0
      ...
      1.0
      1.0
      0.0
      0.0
      definite
      definite
      NaN
      81.594055
      female
      1

      6
      81.601997
      81.771004
      62.0
      7.189999
      NaN
      white
      940.0
      28.0
      born at Duke
      0.0
      ...
      1.0
      0.0
      0.0
      0.0
      absent
      absent
      absent
      81.602295
      female
      0

      13
      81.683998
      81.853996
      62.0
      7.179996
      182.0
      black
      1110.0
      28.0
      born at Duke
      0.0
      ...
      0.0
      1.0
      0.0
      1.0
      absent
      absent
      absent
      81.684448
      male
      0

      14
      81.689003
      81.877998
      69.0
      7.419998
      361.0
      white
      1180.0
      28.0
      born at Duke
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      absent
      absent
      absent
      81.689880
      male
      0

      16
      81.696999
      81.952004
      93.0
      7.239998
      255.0
      black
      770.0
      26.0
      born at Duke
      0.0
      ...
      1.0
      0.0
      0.0
      1.0
      absent
      absent
      absent
      81.698120
      male
      0

5 rows × 26 columns

To demonstrate binary data analysis, we will try to estimate the difference between the probability of an intra-ventricular hemorrhage for infants with a pneumothorax.

```python
pd.crosstab(vlbw.ivh, vlbw.pneumo)
```

      pneumo
      0.0
      1.0

      ivh

      absent
      359
      73

      definite
      45
      30

      possible
      6
      4

We will create a binary outcome by combining `definite` and `possible` into a single outcome.

```python
ivh = vlbw.ivh.isin(['definite', 'possible']).astype(int).values
x = vlbw.pneumo.astype(int).values
```

### Prior choice

What should we choose as a prior distribution for $p$?

We could stick with a normal distribution, but note that the value of $p$ is **constrained** by the laws of probability. Namely, we cannot have values smaller than zero nor larger than one. So, choosing a normal distribution will result in ascribing positive probability to unsupported values of the parameter. In many cases, this will still work in practice, but will be inefficient for calculating the posterior and will not accurately represent the prior information about the parameter.

A common choice in this context is the **beta distribution**, a continuous distribution with 2 parameters and whose support is on the unit interval:

$$ f(x \mid \alpha, \beta) = \frac{x^{\alpha - 1} (1 - x)^{\beta - 1}}{B(\alpha, \beta)}$$

- Support: $x \in (0, 1)$
- Mean: $\dfrac{\alpha}{\alpha + \beta}$
- Variance: $\dfrac{\alpha \beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$

```python
from pymc3 import Beta

params = (5, 1), (1, 3), (5, 5), (0.5, 0.5), (1, 1)

fig, axes = plt.subplots(1, len(params), figsize=(14, 4), sharey=True)
for ax, (alpha, beta) in zip(axes, params):
    sns.distplot(Beta.dist(alpha, beta).random(size=10000), ax=ax, kde=False)
    ax.set_xlim(0, 1)
    ax.set_title(r'$\alpha={0}, \beta={1}$'.format(alpha, beta));
```

So let's use a beta distribution to model our prior knowledge of the probabilities for both groups. Setting $\alpha = \beta = 1$ will result in a uniform distribution of prior mass:

```python
with Model() as ivh_model:

    p = Beta('p', 1, 1, shape=2)
```

We can now use `p` as the parameter of our Bernoulli likelihood. Here, `x` is a vector of zeros an ones, which will extract the approproate group probability for each subject:

```python
from pymc3 import Bernoulli

with ivh_model:

    bb_like = Bernoulli('bb_like', p=p[x], observed=ivh)
```

Finally, since we are interested in the difference between the probabilities, we will keep track of this difference:

```python
with ivh_model:

    p_diff = Deterministic('p_diff', p[1] - p[0])
```

```python
with ivh_model:
    ivh_trace = sample(1000, init=None)
```

    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [p]
    Sampling 2 chains: 100%|██████████| 3000/3000 [00:01<00:00, 1636.77draws/s]

```python
plot_posterior(ivh_trace[100:], varnames=['p'], color='#87ceeb');
```

We can see that the probability that `p` is larger for the pneumothorax with probability one.

```python
plot_posterior(ivh_trace[100:], varnames=['p_diff'], ref_val=0, color='#87ceeb');
```

As a final demonstration, let's make this analysis slightly more realistic. Since these are observational data, its probably a bad idea to compare group probabilities without correcting them for **confounding variables**. Individuals may have systematic differences between groups that may explain the difference in observed outcomes that have nothing to do with group membership itself.

For example, perhaps lower birthweight accounts for the difference, since birthweight is nominally lower in the pneumothorax group:

```python
vlbw.groupby('pneumo').bwt.mean()
```

    pneumo
    0.0    1089.395122
    1.0    1035.355140
    Name: bwt, dtype: float64

Let's add birth weight as a predictor to our model (but first we will center it to aid interpretation):

```python
bwt_centered = (vlbw.bwt - vlbw.bwt.mean()) / vlbw.bwt.std()
```

How do we add a continuous covariate to this model?

The easiest way to do so is to change the formulation of the model to that of a **generalized linear model (GLM)**. What this does is create a regression model to predict the latent probability of IVH for each individual.

$$p_i = \mu + \alpha * \text{bwt}_i + \beta * \text{pneumo}_i$$

here, $\mu$ is a baseline probability, $\alpha$ is a coefficient for centered birthweight, and $\beta$ is a coefficient for pneumothorax.

However, this presents a problem as formulated: its very easy to generate combinations of $\mu$, $\alpha$, and $\beta$ that generate values outside the $(0,1)$ interval. To avoid this, we must **transform** this sum in such a way that the unit interval constraint is guaranteed.

One such transformation is the **logit**:

$$x = \text{logit}(p) = \log\left[\frac{p}{1-p}\right]$$

the inverse of this function maps real values of $x$ to values of $p$ on the unit interval:

$$p = \text{invlogit}(x) = \frac{1}{1 + \exp(-x)}$$

```python
x_range = np.linspace(-7,7)
plt.plot(x_range, 1/(1+np.exp(-x_range)))
plt.xlabel('x')
plt.ylabel('p');
```

To convert our original model to a GLM, we need to specify priors for the linear model coefficients:

```python
with Model() as ivh_glm:

    μ = Normal('μ', 0, sd=10)
    α = Normal('α', 0, sd=10)
    β = Normal('β', 0, sd=10)
```

Then, apply the `invlogit` transformation to the linear combination of predictors.

The likelihood stays the same.

```python
from pymc3.math import invlogit

with ivh_glm:

    p = invlogit(μ + α*bwt_centered + β*x)

    bb_like = Bernoulli('bb_like', p=p, observed=ivh)
```

```python
with ivh_glm:
    trace_glm = sample(1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [β, α, μ]
    Sampling 2 chains: 100%|██████████| 3000/3000 [00:03<00:00, 983.48draws/s]

We can see that the estimate for $\alpha$ is negative, indicating that higher birth weight reduces the baseline probability of IVH.

The coefficient for $\beta$ is strongly positive.

```python
plot_posterior(trace_glm, varnames=['μ', 'α', 'β'],
               color='#87ceeb');
```

---
## To Learn More

- Kruschke, J.K. [Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan](https://www.amazon.com/Doing-Bayesian-Data-Analysis-Tutorial/dp/0124058884). 2015. Academic Press / Elsevier.
- [Probabilistic Programming and Bayesian Methods for Hackers](http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/) by Cam Davidson Pilon
- [PyMC3 Tutorial Notebooks](https://docs.pymc.io/nb_tutorials/index.html)


<center>

## [mlcourse.ai](mlcourse.ai) – Open Machine Learning Course
### <center> Author: Ilya Larchenko, ODS Slack ilya_l

## <center> Individual data analysis project

## 1. Data description

__I will analyse California Housing Data (1990). It can be downloaded from Kaggle [https://www.kaggle.com/harrywang/housing]__

We will predict the median price of household in block.
To start you need to download file housing.csv.zip . Let's load the data and look at it.

```python
import pandas as pd
import numpy as np
import os
%matplotlib inline

import warnings                                  # `do not disturbe` mode
warnings.filterwarnings('ignore')
```

```python
# change this if needed
PATH_TO_DATA = 'data'
```

```python
full_df = pd.read_csv(os.path.join(PATH_TO_DATA, 'housing.csv.zip'), compression ='zip')
print(full_df.shape)
full_df.head()
```

Data consists of 20640 rows and 10 features:

1. longitude: A measure of how far west a house is; a higher value is farther west
2. latitude: A measure of how far north a house is; a higher value is farther north
3. housingMedianAge: Median age of a house within a block; a lower number is a newer building
4. totalRooms: Total number of rooms within a block
5. totalBedrooms: Total number of bedrooms within a block
6. population: Total number of people residing within a block
7. households: Total number of households, a group of people residing within a home unit, for a block
8. medianIncome: Median income for households within a block of houses (measured in tens of thousands of US Dollars)
9. medianHouseValue: Median house value for households within a block (measured in US Dollars)
10. oceanProximity: Location of the house w.r.t ocean/sea

*median_house_value* is our target feature, we will use other features to predict it.

The task is to predict how much the houses in particular block cost (the median) based on information of blocks location and basic sociodemographic data

Let's divide dataset into train (75%) and test (25%).

```python
%%time
from sklearn.model_selection import train_test_split
train_df, test_df = train_test_split(full_df,shuffle = True, test_size = 0.25, random_state=17)
train_df=train_df.copy()
test_df=test_df.copy()
print(train_df.shape)
print(test_df.shape)
```

All futher analysis we will do with the test set. But feature generation and processing will be simmultaneously done on both sets.

## 2-3. Primary data analysis / Primary visual data analysis

```python
train_df.describe()
```

```python
train_df.info()
```

We can see that most columns has no nan values (except total_bedrooms), most features has float format, only 1 feature is categorical - ocean_proximity.

```python
train_df[pd.isnull(train_df).any(axis=1)].head(10)
```

There is no obvious reasons for some total_bedrooms to be NaN. The number of NaNs is about 1% of total dataset. Maybe we could just drop this rows or fill it with mean/median values, but let's wait for a while, and deal with blanks after initial data analysis in a smarter manner.

Let's create the list of numeric features names (it will be useful later).

```python
numerical_features=list(train_df.columns)
numerical_features.remove('ocean_proximity')
numerical_features.remove('median_house_value')
print(numerical_features)
```

Let's look at target feature distribition

```python
train_df['median_house_value'].hist()
```

We can visually see that distribution is skewed and not normal. Also it seems that the values are clipped somewhere near 500 000. We can check it numerically.

```python
max_target=train_df['median_house_value'].max()
print("The largest median value:",max_target)
print("The # of values, equal to the largest:", sum(train_df['median_house_value']==max_target))
print("The % of values, equal to the largest:", sum(train_df['median_house_value']==max_target)/train_df.shape[0])
```

Almost 5% of all values = exactly 500 001. It proves our clipping theory. Let's check the clipping of small values:

```python
min_target=train_df['median_house_value'].min()
print("The smallest median value:",min_target)
print("The # of values, equal to the smallest:", sum(train_df['median_house_value']==min_target))
print("The % of values, equal to the smallest:", sum(train_df['median_house_value']==min_target)/train_df.shape[0])
```

This time it looks much better, a little bit artificial value 14 999 - is common for prices. And there are only 4 such values. So probably the small values are not clipped.

Let's conduct some normality tests:

```python
from statsmodels.graphics.gofplots import qqplot
from matplotlib import pyplot

qqplot(train_df['median_house_value'], line='s')
pyplot.show()
```

```python
from scipy.stats import normaltest

stat, p = normaltest(train_df['median_house_value'])
print('Statistics=%.3f, p=%.3f' % (stat, p))

alpha = 0.05
if p < alpha:  # null hypothesis: x comes from a normal distribution
    print("The null hypothesis can be rejected")
else:
    print("The null hypothesis cannot be rejected")
```

QQ-plot and D’Agostino and Pearson’s normality test show that the distribution is far from normal. We can try to use log(1+n) to make it more normal:

```python
target_log=np.log1p(train_df['median_house_value'])
qqplot(target_log, line='s')
pyplot.show()
```

```python
stat, p = normaltest(target_log)
print('Statistics=%.3f, p=%.3f' % (stat, p))

alpha = 0.05
if p < alpha:  # null hypothesis: x comes from a normal distribution
    print("The null hypothesis can be rejected")
else:
    print("The null hypothesis cannot be rejected")
```

This graph looks much better, the only non-normal parts are clipped high prices and very low prices. Unfortunately we can not reconstruct clipped data and statistically the distribution it is still not normal - p-value = 0, the null hypothesis of distribution normality can be rejected.

Anyway, predicting of target_log instead of target can be a good choice for us, but we still should check it during model validation phase.

```python
train_df['median_house_value_log']=np.log1p(train_df['median_house_value'])
test_df['median_house_value_log']=np.log1p(test_df['median_house_value'])
```

Now let's analyze numerical features. First of all we need to look at their distributions.

```python
train_df[numerical_features].hist(bins=50, figsize=(10, 10))
```

Some features are signifacantly skewed, and our "log trick" should be heplfull

```python
skewed_features=['households','median_income','population', 'total_bedrooms', 'total_rooms']
log_numerical_features=[]
for f in skewed_features:
    train_df[f + '_log']=np.log1p(train_df[f])
    test_df[f + '_log']=np.log1p(test_df[f])
    log_numerical_features.append(f + '_log')
```

```python
train_df[log_numerical_features].hist(bins=50, figsize=(10, 10))
```

Our new features looks much better (during the modeling phase we can use either original, new ones or both of them)

housing_median_age looks clipped as well. Let's look at it's highest value precisely.

```python
max_house_age=train_df['housing_median_age'].max()
print("The largest value:",max_house_age)
print("The # of values, equal to the largest:", sum(train_df['housing_median_age']==max_house_age))
print("The % of values, equal to the largest:", sum(train_df['housing_median_age']==max_house_age)/train_df.shape[0])
```

It is very likely the data is clipped (there are also a small chance that in 1938 there was a great reconstruction project in California but it seems less likely). We can't recreate original values, but it can be useful to create new binary value indicating the clipping of the house age.

```python
train_df['age_clipped']=train_df['housing_median_age']==max_house_age
test_df['age_clipped']=test_df['housing_median_age']==max_house_age
```

Now we will analyse correleation between features and target variable

```python
import matplotlib.pyplot as plt
import seaborn as sns

corr_y = pd.DataFrame(train_df).corr()
plt.rcParams['figure.figsize'] = (20, 16)  # Размер картинок
sns.heatmap(corr_y,
            xticklabels=corr_y.columns.values,
            yticklabels=corr_y.columns.values, annot=True)
```

We can see some (maybe obvious) patterns here:
    - House values are significantly correlated with median income
    - Number of households is not 100% correlated with population, we can try to add average_size_of_household as a feature
    - Longitude and Latitude should be analyzed separately (just a correlation with target variable is not very useful)
    - There is a set of highly correlated features: number of rooms, bedrooms, population and households. It can be useful to reduce dimensionality of this subset, especially if we use linear models
    - total_bedrooms is one of these highly correlated features, it means we can fill NaN values with high precision using simplest linear regression

Let's try to fill NaNs with simple linear regression:

```python
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

lin = LinearRegression()

# we will train our model based on all numerical non-target features with not NaN total_bedrooms
appropriate_columns = train_df.drop(['median_house_value','median_house_value_log',
                            'ocean_proximity', 'total_bedrooms_log'],axis=1)
train_data=appropriate_columns[~pd.isnull(train_df).any(axis=1)]

# model will be validated on 25% of train dataset
# theoretically we can use even our test_df dataset (as we don't use target) for this task, but we will not
temp_train, temp_valid = train_test_split(train_data,shuffle = True, test_size = 0.25, random_state=17)

lin.fit(temp_train.drop(['total_bedrooms'],axis=1), temp_train['total_bedrooms'])
np.sqrt(mean_squared_error(lin.predict(temp_valid.drop(['total_bedrooms'],axis=1)),
                           temp_valid['total_bedrooms']))
```

RMSE on a validation set is 64.5. Let's compare this with the best constant prediction - what if we fill NaNs with mean value:

```python
np.sqrt(mean_squared_error(np.ones(len(temp_valid['total_bedrooms']))*temp_train['total_bedrooms'].mean(),
                           temp_valid['total_bedrooms']))
```

Obviously our linear regression approach is much better. Let's train our model on whole train dataset and apply it to the rows with blanks. But preliminary we will "remember" the rows with NaNs, because there is a chance, that it can contain useful information.

```python
lin.fit(train_data.drop(['total_bedrooms'],axis=1), train_data['total_bedrooms'])

train_df['total_bedrooms_is_nan']=pd.isnull(train_df).any(axis=1).astype(int)
test_df['total_bedrooms_is_nan']=pd.isnull(test_df).any(axis=1).astype(int)

train_df['total_bedrooms'].loc[pd.isnull(train_df).any(axis=1)]=\
lin.predict(train_df.drop(['median_house_value','median_house_value_log','total_bedrooms','total_bedrooms_log',
               'ocean_proximity','total_bedrooms_is_nan'],axis=1)[pd.isnull(train_df).any(axis=1)])

test_df['total_bedrooms'].loc[pd.isnull(test_df).any(axis=1)]=\
lin.predict(test_df.drop(['median_house_value','median_house_value_log','total_bedrooms','total_bedrooms_log',
               'ocean_proximity','total_bedrooms_is_nan'],axis=1)[pd.isnull(test_df).any(axis=1)])

#linear regression can lead to negative predictions, let's change it
test_df['total_bedrooms']=test_df['total_bedrooms'].apply(lambda x: max(x,0))
train_df['total_bedrooms']=train_df['total_bedrooms'].apply(lambda x: max(x,0))
```

Let's update 'total_bedrooms_log' and check if there are no NaNs left

```python
train_df['total_bedrooms_log']=np.log1p(train_df['total_bedrooms'])
test_df['total_bedrooms_log']=np.log1p(test_df['total_bedrooms'])
```

```python
print(train_df.info())
print(test_df.info())
```

After filling of blanks let's have a closer look on dependences between some numerical features

```python
sns.set()
sns.pairplot(train_df[log_numerical_features+['median_house_value_log']])
```

It seems there are no new insights about numerical features (only confirmation of the old ones).

Let's try to do the same thing but for the local (geographically) subset of our data.

```python
sns.set()
local_coord=[-122, 41] # the point near which we want to look at our variables
euc_dist_th = 2 # distance treshhold

euclid_distance=train_df[['latitude','longitude']].apply(lambda x:
                                                         np.sqrt((x['longitude']-local_coord[0])**2+
                                                                 (x['latitude']-local_coord[1])**2), axis=1)

# indicate wethere the point is within treshhold or not
indicator=pd.Series(euclid_distance<=euc_dist_th, name='indicator')

print("Data points within treshhold:", sum(indicator))

# a small map to visualize th eregion for analysis
sns.lmplot('longitude', 'latitude', data=pd.concat([train_df,indicator], axis=1), hue='indicator', markers ='.', fit_reg=False, height=5)

# pairplot
sns.pairplot(train_df[log_numerical_features+['median_house_value_log']][indicator])
```

We can see that on any local territory (you can play with local_coord and euc_dist_th) the linear dependences between variables became stronger, especially median_income_log / median_house_value_log. So the coordinates is very important factor for our task (we will analyze it later)

Now let's move on to the categorical feature "ocean_proximity". It is not 100% clear what does it values means. So let's first of all plot in on the map.

```python
sns.lmplot('longitude', 'latitude', data=train_df,markers ='.', hue='ocean_proximity', fit_reg=False, height=5)
plt.show()
```

Now we better undersand the meaning of different classes. Let's look at the data.

```python
value_count=train_df['ocean_proximity'].value_counts()
value_count
```

```python
plt.figure(figsize=(12,5))

sns.barplot(value_count.index, value_count.values)
plt.title('Ocean Proximity')
plt.ylabel('Number of Occurrences')
plt.xlabel('Ocean Proximity')

plt.figure(figsize=(12,5))
plt.title('House Value depending on Ocean Proximity')
sns.boxplot(x="ocean_proximity", y="median_house_value_log", data=train_df)
```

We can see that INLAND houses has significantly lower prices. Distribution in other differ but not so much. There is no clear trend in house price / poximity, so we will not try to invent complex encoding approach. Let's just do OHE for this feature.

```python
ocean_proximity_dummies = pd.get_dummies(pd.concat([train_df['ocean_proximity'],test_df['ocean_proximity']]),
                                         drop_first=True)
```

```python
dummies_names=list(ocean_proximity_dummies.columns)
```

```python
train_df=pd.concat([train_df,ocean_proximity_dummies[:train_df.shape[0]]], axis=1 )
test_df=pd.concat([test_df,ocean_proximity_dummies[train_df.shape[0]:]], axis=1 )

train_df=train_df.drop(['ocean_proximity'], axis=1)
test_df=test_df.drop(['ocean_proximity'], axis=1)
```

```python
train_df.head()
```

And finally we will explore coordinates features.

```python
train_df[['longitude','latitude']].describe()
```

Let's plot the house_values (target) on map:

```python
from matplotlib.colors import LinearSegmentedColormap

plt.figure(figsize=(10,10))

cmap = LinearSegmentedColormap.from_list(name='name', colors=['green','yellow','red'])

f, ax = plt.subplots()
points = ax.scatter(train_df['longitude'], train_df['latitude'], c=train_df['median_house_value_log'],
                    s=10, cmap=cmap)
f.colorbar(points)
```

It seems that the average value of geographically nearest houses can be very good feature.

We can also see, that the most expensive houses are located near San Francisco (37.7749° N, 122.4194° W) and Los Angeles (34.0522° N, 118.2437°). Based on this we can use the distance to this cities as additional features.

We also see that the most expensive houses are on approximately on the straight line, and become cheaper when we moving to North-East. This means that the linear combination of coordinates themselves can be useful feature as well.

```python
sf_coord=[-122.4194, 37.7749]
la_coord=[-118.2437, 34.0522]

train_df['distance_to_SF']=np.sqrt((train_df['longitude']-sf_coord[0])**2+(train_df['latitude']-sf_coord[1])**2)
test_df['distance_to_SF']=np.sqrt((test_df['longitude']-sf_coord[0])**2+(test_df['latitude']-sf_coord[1])**2)

train_df['distance_to_LA']=np.sqrt((train_df['longitude']-la_coord[0])**2+(train_df['latitude']-la_coord[1])**2)
test_df['distance_to_LA']=np.sqrt((test_df['longitude']-la_coord[0])**2+(test_df['latitude']-la_coord[1])**2)
```

## 4. Insights and found dependencies

Let's quickly sum up what useful we have found so far:
- We have analyzed the features and found some ~lognorm distributed among them. We have created corresponding log features
- We have analyzed the distribution of the target feature, and concluded that it may be useful to predict log of it (to be checked)
- We have dealt with clipped and missing data
- We have created features corresponding to simple Eucledian distances to LA ans SF
- We also has found several highly correlated variables and maybe will work with them later
- We have already generated several new variables and will create more of them later after the initial modeling phase

All explanation about this steps were already given above.

## 5. Metrics selection

This is regression problem. Our target metric will be RMSE - it is one of the most popular regression metrics, and it has same unit of measurement as target value thus is easy to explain to other people.

\begin{align}
RMSE = \sqrt{\frac{1}{n}\Sigma_{i=1}^{n}{\Big(\frac{d_i -f_i}{\sigma_i}\Big)^2}}
\end{align}

As far as there is a monotonic dependence between RMSE and MSE we can optimize MSE in our model and compute RMSE only in the end. MSE is easy to optimize it is a default loss function for the most of regression models.

The main drawback of MSE and RMSE - high penalty for big errors in predictions - it can overfit to outliers, but in our case outlaying target values have already been clipped so it is not a big problem.

## 6. Model selection

We will try to solve our problem with 3 different regression models:
- Linear regression
- Random forest
- Gradient boosting

Linear regression is fast, simple and can provide quite a good baseline result for our task.
Tree based models can provide better results in case of nonlinear complex dependences of variables and in case of small number of variables, they are also more stable to multicollinearity (and we have highly correlated variables). Moreover in our problem target values are clipped and targets can't be outside the clipping interval, it is good for the tree-based models.

The results of using these models will be compared in the 11-12 parts of the project. Tree-based models are expected to work better in this particular problem, but we will start with more simple model.

We will start with standard linear regression, go through all of the modeling steps, and then do some simplified computations for 2 other models (without in-depth explanation of every step).

The final model selection will be done based on the results.

## 7. Data preprocessing

We have already done most of the preprocessing steps:
    - OHE for the categorical features
    - Filled NaNs
    - Computed logs of skewed data
    - Divided data into train and hold-out sets

Now let's scale all numerical features (it is useful for the linear models), prepare cross validation splits and we are ready to proceed to modeling

```python
from sklearn.preprocessing import StandardScaler

features_to_scale=numerical_features+log_numerical_features+['distance_to_SF','distance_to_LA']

scaler = StandardScaler()

X_train_scaled=pd.DataFrame(scaler.fit_transform(train_df[features_to_scale]),
                            columns=features_to_scale, index=train_df.index)
X_test_scaled=pd.DataFrame(scaler.transform(test_df[features_to_scale]),
                           columns=features_to_scale, index=test_df.index)
```

## 8 Cross-validation and adjustment of model hyperparameters

Let's prepare cross validation samples.
As far as there are not a lot of data we can easily divide it on 10 folds, that are taken from shuffled train data.
Within every split we will train our model on 90% of train data and compute CV metric on the other 10%.

We fix the random state for the reproducibility.

```python
from sklearn.model_selection import KFold, cross_val_score

kf = KFold(n_splits=10, random_state=17, shuffle=True)
```

### Linear regression

For the first initial baseline we will take Rigge model with only initial numerical and OHE features

```python
from sklearn.linear_model import Ridge

model=Ridge(alpha=1)
X=train_df[numerical_features+dummies_names]
y=train_df['median_house_value']
cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
print(np.sqrt(-cv_scores.mean()))
```

We are doing cross validation with 10 folds, computing 'neg_mean_squared_error' (neg - because sklearn needs scoring functions to be minimized). Our final metrics: RMSE=np.sqrt(-neg_MSE)

So our baseline is RMSE = $68 702 we will try to improve this results using everything we have discovered during the data analysis phase.

We will do the following steps:
    - Use scaled features
    - Add log features
    - Add NaN and age clip indicating features
    - Add city-distance features
    - Generate several new features
    - Try to predict log(target) instead of target
    - Tune some hyperparameters of the model

One again the most part of the hyperparameters adjustment will be done later after we add some new features. Actually the cross-validation and parameters tuning process is done through the parts 8-11.

```python
# using scaled data
X=pd.concat([train_df[dummies_names], X_train_scaled[numerical_features]], axis=1, ignore_index = True)
cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
print(np.sqrt(-cv_scores.mean()))
```

```python
# adding NaN indicating feature
X=pd.concat([train_df[dummies_names+['total_bedrooms_is_nan']],
             X_train_scaled[numerical_features]], axis=1, ignore_index = True)
cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
print(np.sqrt(-cv_scores.mean()))
```

```python
# adding house age cliiping indicating feature
X=pd.concat([train_df[dummies_names+['age_clipped']],
             X_train_scaled[numerical_features]], axis=1, ignore_index = True)
cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
print(np.sqrt(-cv_scores.mean()))
```

```python
# adding log features
X=pd.concat([train_df[dummies_names+['age_clipped']], X_train_scaled[numerical_features+log_numerical_features]],
            axis=1, ignore_index = True)
cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
print(np.sqrt(-cv_scores.mean()))
```

```python
# adding city distance features
X=pd.concat([train_df[dummies_names+['age_clipped']], X_train_scaled],
            axis=1, ignore_index = True)
cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
print(np.sqrt(-cv_scores.mean()))
```

Up to this moment we have got best result using numerical features + their logs + age_clipped+ dummy variables + distances to the largest cities.
Let's try to generate new features

## 9. Creation of new features and description of this process

Previously we have already created and explained the rational of new features creation. Now Let's generate additional ones

City distances features work, but maybe there are also some non-linear dependencies between them and the target variables.

```python
sns.set()
sns.pairplot(train_df[['distance_to_SF','distance_to_LA','median_house_value_log']])
```

Visually is not obvious so let's try to create a couple of new variables and check:

```python
new_features_train_df=pd.DataFrame(index=train_df.index)
new_features_test_df=pd.DataFrame(index=test_df.index)

new_features_train_df['1/distance_to_SF']=1/(train_df['distance_to_SF']+0.001)
new_features_train_df['1/distance_to_LA']=1/(train_df['distance_to_LA']+0.001)
new_features_train_df['log_distance_to_SF']=np.log1p(train_df['distance_to_SF'])
new_features_train_df['log_distance_to_LA']=np.log1p(train_df['distance_to_LA'])

new_features_test_df['1/distance_to_SF']=1/(test_df['distance_to_SF']+0.001)
new_features_test_df['1/distance_to_LA']=1/(test_df['distance_to_LA']+0.001)
new_features_test_df['log_distance_to_SF']=np.log1p(test_df['distance_to_SF'])
new_features_test_df['log_distance_to_LA']=np.log1p(test_df['distance_to_LA'])
```

We can also generate some features correlated to the prosperity:
- rooms/person - how many rooms are there per person. The higher - the richer people are living there - the more expensive houses they buy
- rooms/household - how many rooms are there per family. The similar one but corresponds to the number of rooms per family (assuming household~family), not per person.
- two similar features but counting only bedrooms

```python
new_features_train_df['rooms/person']=train_df['total_rooms']/train_df['population']
new_features_train_df['rooms/household']=train_df['total_rooms']/train_df['households']

new_features_test_df['rooms/person']=test_df['total_rooms']/test_df['population']
new_features_test_df['rooms/household']=test_df['total_rooms']/test_df['households']

new_features_train_df['bedrooms/person']=train_df['total_bedrooms']/train_df['population']
new_features_train_df['bedrooms/household']=train_df['total_bedrooms']/train_df['households']

new_features_test_df['bedrooms/person']=test_df['total_bedrooms']/test_df['population']
new_features_test_df['bedrooms/household']=test_df['total_bedrooms']/test_df['households']
```

- the luxurity of house can be characterized buy number of bedrooms per rooms

```python
new_features_train_df['bedroom/rooms']=train_df['total_bedrooms']/train_df['total_rooms']
new_features_test_df['bedroom/rooms']=test_df['total_bedrooms']/test_df['total_rooms']
```

- the average number of persons in one household can be the signal of prosperity or the same time the signal of richness but in any case it can be a useful feature

```python
new_features_train_df['average_size_of_household']=train_df['population']/train_df['households']
new_features_test_df['average_size_of_household']=test_df['population']/test_df['households']
```

And finally let's scale all this features

```python
new_features_train_df=pd.DataFrame(scaler.fit_transform(new_features_train_df),
                            columns=new_features_train_df.columns, index=new_features_train_df.index)

new_features_test_df=pd.DataFrame(scaler.transform(new_features_test_df),
                            columns=new_features_test_df.columns, index=new_features_test_df.index)
```

```python
new_features_train_df.head()
```

```python
new_features_test_df.head()
```

We will add new features one by one and keeps only those that improve our best score

```python
# computing current best score

X=pd.concat([train_df[dummies_names+['age_clipped']], X_train_scaled],
            axis=1, ignore_index = True)

cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
best_score = np.sqrt(-cv_scores.mean())
print("Best score: ", best_score)

# list of the new good features
new_features_list=[]

for feature in new_features_train_df.columns:
    new_features_list.append(feature)
    X=pd.concat([train_df[dummies_names+['age_clipped']], X_train_scaled,
                 new_features_train_df[new_features_list]
                ],
                axis=1, ignore_index = True)
    cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
    score = np.sqrt(-cv_scores.mean())
    if score >= best_score:
        new_features_list.remove(feature)
        print(feature, ' is not a good feature')
    else:
        print(feature, ' is a good feature')
        print('New best score: ', score)
        best_score=score
```

We have got 5 new good features. Let's update our X variable

```python
X=pd.concat([train_df[dummies_names+['age_clipped']], X_train_scaled,
             new_features_train_df[new_features_list]
            ],
            axis=1).reset_index(drop=True)
y=train_df['median_house_value'].reset_index(drop=True)
```

To deal with log of target we need to create our own cross validation or our own predicting model. We will try the first option

```python
from sklearn.metrics import mean_squared_error

def cross_val_score_with_log(model=model, X=X,y=y,kf=kf, use_log=False):

    X_temp=np.array(X)

    # if use_log parameter is true we will predict log(y+1)
    if use_log:
        y_temp=np.log1p(y)
    else:
        y_temp=np.array(y)

    cv_scores=[]
    for train_index, test_index in kf.split(X_temp,y_temp):

        prediction = model.fit(X_temp[train_index], y_temp[train_index]).predict(X_temp[test_index])

        # if use_log parameter is true we should come back to the initial targer
        if use_log:
            prediction=np.expm1(prediction)
        cv_scores.append(-mean_squared_error(y[test_index],prediction))

    return np.sqrt(-np.mean(cv_scores))
```

```python
cross_val_score_with_log(X=X,y=y,kf=kf, use_log=False)
```

We have got exactly the same result as with cross_val_score function. That means everything work ok. Now let's try to set use_log to true

```python
cross_val_score_with_log(X=X,y=y,kf=kf, use_log=True)
```

Unfortunately, it has not helped. So we will stick to the previous version.

And now we will tune the only meaningful hyperparameter of the Ridge regression - alpha.

## 10. Plotting training and validation curves

Let's plot Validation Curve

```python
from sklearn.model_selection import validation_curve

Cs=np.logspace(-5, 4, 10)
train_scores, valid_scores = validation_curve(model, X, y, "alpha",
                                              Cs, cv=kf, scoring='neg_mean_squared_error')

plt.plot(Cs, np.sqrt(-train_scores.mean(axis=1)), 'ro-')

plt.fill_between(x=Cs, y1=np.sqrt(-train_scores.max(axis=1)),
                 y2=np.sqrt(-train_scores.min(axis=1)), alpha=0.1, color = "red")

plt.plot(Cs, np.sqrt(-valid_scores.mean(axis=1)), 'bo-')

plt.fill_between(x=Cs, y1=np.sqrt(-valid_scores.max(axis=1)),
                 y2=np.sqrt(-valid_scores.min(axis=1)), alpha=0.1, color = "blue")

plt.xscale('log')
plt.xlabel('alpha')
plt.ylabel('RMSE')
plt.title('Regularization Parameter Tuning')

plt.show()
```

```python
Cs[np.sqrt(-valid_scores.mean(axis=1)).argmin()]
```

We can see that curves for train and CV are very close to each other, it is a sign of underfiting. The difference between the curves does not change along with change in alpha this mean that we should try more complex models comparing to linear regression or add more new features (f.e. polynomial ones)

Using this curve we can find the optimal value of alpha. It is alpha=1. But actually our prediction does not change when alpha goes below 1.

Let's use alpha=1 and plot the learning curve

```python
from sklearn.model_selection import learning_curve

model=Ridge(alpha=1.0)

train_sizes, train_scores, valid_scores = learning_curve(model, X, y, train_sizes=list(range(50,10001,100)),
                                                         scoring='neg_mean_squared_error', cv=5)

plt.plot(train_sizes, np.sqrt(-train_scores.mean(axis=1)), 'ro-')

plt.fill_between(x=train_sizes, y1=np.sqrt(-train_scores.max(axis=1)),
                 y2=np.sqrt(-train_scores.min(axis=1)), alpha=0.1, color = "red")

plt.plot(train_sizes, np.sqrt(-valid_scores.mean(axis=1)), 'bo-')

plt.fill_between(x=train_sizes, y1=np.sqrt(-valid_scores.max(axis=1)),
                 y2=np.sqrt(-valid_scores.min(axis=1)), alpha=0.1, color = "blue")

plt.xlabel('Train size')
plt.ylabel('RMSE')
plt.title('Regularization Parameter Tuning')

plt.show()
```

Learning curves indicate high bias of the model - this means we will not improve our model by adding more data, but we can try to use more complex models or add more features to improve the results.

This result is inline with the validation curve results. So let's move on to the more complex models.

### Random forest

Actually we can just put all our features into the model but we can easily improve computational performance of the tree-based models, by deleting all monotonous derivatives of features because they does not help at all.

For example, adding log(feature) don't help tree-based model, it will just make it more computationally intensive.

So let's train random forest classifier based on shorten set of the features

```python
X.columns
```

```python
features_for_trees=['INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN', 'age_clipped',
       'longitude', 'latitude', 'housing_median_age', 'total_rooms',
       'total_bedrooms', 'population', 'households', 'median_income',
       'distance_to_SF', 'distance_to_LA','bedroom/rooms']
```

```python
%%time
from sklearn.ensemble import RandomForestRegressor

X_trees=X[features_for_trees]

model_rf=RandomForestRegressor(n_estimators=100, random_state=17)
cv_scores = cross_val_score(model_rf, X_trees, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)

print(np.sqrt(-cv_scores.mean()))
```

We can see significant improvement, comparing to the linear model and higher n_estimator probably will help. But first, let's try to tune other hyperparametres:

```python
from sklearn.model_selection import GridSearchCV

param_grid={'n_estimators': [100],
            'max_depth':  [22, 23, 24, 25],
            'max_features': [5,6,7,8]}

gs=GridSearchCV(model_rf, param_grid, scoring='neg_mean_squared_error', fit_params=None, n_jobs=-1, cv=kf, verbose=1)

gs.fit(X_trees,y)
```

```python
print(np.sqrt(-gs.best_score_))
```

```python
gs.best_params_
```

```python
best_depth=gs.best_params_['max_depth']
best_features=gs.best_params_['max_features']
```

```python
%%time
model_rf=RandomForestRegressor(n_estimators=100, max_depth=best_depth, max_features=best_features, random_state=17)
cv_scores = cross_val_score(model_rf, X_trees, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)

print(np.sqrt(-cv_scores.mean()))
```

With the relatively small effort we have got a significant improvement of results. Random Forest results can be further improved by higher n_estimators, let's find the n_estimators at witch the results stabilize.

```python
model_rf=RandomForestRegressor(n_estimators=200,  max_depth=best_depth, max_features=best_features, random_state=17)
Cs=list(range(20,201,20))
train_scores, valid_scores = validation_curve(model_rf, X_trees, y, "n_estimators",
                                              Cs, cv=kf, scoring='neg_mean_squared_error')

plt.plot(Cs, np.sqrt(-train_scores.mean(axis=1)), 'ro-')

plt.fill_between(x=Cs, y1=np.sqrt(-train_scores.max(axis=1)),
                 y2=np.sqrt(-train_scores.min(axis=1)), alpha=0.1, color = "red")

plt.plot(Cs, np.sqrt(-valid_scores.mean(axis=1)), 'bo-')

plt.fill_between(x=Cs, y1=np.sqrt(-valid_scores.max(axis=1)),
                 y2=np.sqrt(-valid_scores.min(axis=1)), alpha=0.1, color = "blue")

plt.xlabel('n_estimators')
plt.ylabel('RMSE')
plt.title('Regularization Parameter Tuning')

plt.show()
```

This time we can see that the results of train is much better than CV, but it is totally ok for the Random Forest.

Higher value of n_estimators (>100) does not help much. Let's stick to the n_estimators=200 - it is high enough but not very computationally intensive.

### Gradient boosting

And finally we will try to use LightGBM to solve our problem.
We will try the model out of the box, and then tune some of its parameters using random search

```python
# uncomment to install if you have not yet
#!pip install lightgbm
```

```python
%%time
from lightgbm.sklearn import LGBMRegressor

model_gb=LGBMRegressor()
cv_scores = cross_val_score(model_gb, X_trees, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=1)

print(np.sqrt(-cv_scores.mean()))
```

LGBMRegressor has much more hyperparameters than previous models. As far as this is educational problem we will not spend a lot of time to tuning all of them. In this case RandomizedSearchCV can give us very good result quite fast, much faster than GridSearch. We will do optimization in 2 steps: model complexity optimization and convergence optimization. Let's do it.

```python
gs
```

```python
# model complexity optimization
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_grid={'max_depth':  randint(6,11),
            'num_leaves': randint(7,127),
            'reg_lambda': np.logspace(-3,0,100),
            'random_state': [17]}

gs=RandomizedSearchCV(model_gb, param_grid, n_iter = 50, scoring='neg_mean_squared_error', fit_params=None,
                n_jobs=-1, cv=kf, verbose=1, random_state=17)

gs.fit(X_trees,y)
```

```python
np.sqrt(-gs.best_score_)
```

```python
gs.best_params_
```

Let's fix n_estimators=500, it is big enough but is not to computationally intensive yet, and find the best value of the learning_rate

```python
# model convergency optimization

param_grid={'n_estimators': [500],
            'learning_rate': np.logspace(-4, 0, 100),
            'max_depth':  [10],
            'num_leaves': [72],
            'reg_lambda': [0.0010722672220103231],
            'random_state': [17]}

gs=RandomizedSearchCV(model_gb, param_grid, n_iter = 20, scoring='neg_mean_squared_error', fit_params=None,
                n_jobs=-1, cv=kf, verbose=1, random_state=17)

gs.fit(X_trees,y)
```

```python
np.sqrt(-gs.best_score_)
```

```python
gs.best_params_
```

We have got the best params for the gradient boosting and will use them for the final prediction.

## 11. Prediction for test or hold-out samples

Lets sum up the results of our project. We will compute RMSE on cross validation and holdout set and compare them.

```python
results_df=pd.DataFrame(columns=['model','CV_results', 'holdout_results'])
```

```python
# hold-out features and target
X_ho=pd.concat([test_df[dummies_names+['age_clipped']], X_test_scaled,
             new_features_test_df[new_features_list]],axis=1).reset_index(drop=True)
y_ho=test_df['median_house_value'].reset_index(drop=True)

X_trees_ho=X_ho[features_for_trees]
```

```python
%%time

#linear model
model=Ridge(alpha=1.0)

cv_scores = cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
score_cv=np.sqrt(-np.mean(cv_scores.mean()))

prediction_ho = model.fit(X, y).predict(X_ho)
score_ho=np.sqrt(mean_squared_error(y_ho,prediction_ho))

results_df.loc[results_df.shape[0]]=['Linear Regression',  score_cv,  score_ho]
```

```python
%%time

#Random Forest
model_rf=RandomForestRegressor(n_estimators=200,  max_depth=23, max_features=5, random_state=17)

cv_scores = cross_val_score(model_rf, X_trees, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
score_cv=np.sqrt(-np.mean(cv_scores.mean()))

prediction_ho = model_rf.fit(X_trees, y).predict(X_trees_ho)
score_ho=np.sqrt(mean_squared_error(y_ho,prediction_ho))

results_df.loc[results_df.shape[0]]=['Random Forest',  score_cv,  score_ho]
```

```python
%%time

#Gradient boosting
model_gb=LGBMRegressor(reg_lambda=0.0010722672220103231, max_depth=10,
                       n_estimators=500, num_leaves=72, random_state=17, learning_rate=0.06734150657750829)
cv_scores = cross_val_score(model_gb, X_trees, y, cv=kf, scoring='neg_mean_squared_error', n_jobs=-1)
score_cv=np.sqrt(-np.mean(cv_scores.mean()))

prediction_ho = model_gb.fit(X_trees, y).predict(X_trees_ho)
score_ho=np.sqrt(mean_squared_error(y_ho,prediction_ho))

results_df.loc[results_df.shape[0]]=['Gradient boosting',  score_cv,  score_ho]
```

```python
results_df
```

It seems we have done quite a good job. Cross validation results are inline with holdout ones. Our best CV model - gradient boosting, turned out to be the best on hold-out dataset as well (and it is also faster than random forest).

## 12. Conclusions

To sum up, we have got the solution that can predict the mean house value in the block with RMSE \$46k using our best model - LGB. It is not an extremely precise prediction: \$46k is about 20% of the average mean house price, but it seems that it is near the possible solution for these classes of model based on this data (it is popular dataset but I have not find any solution with significantly better results).

We have used old Californian data from 1990 so it is not useful right now. But the same approach can be used to predict modern house prices (if applied to the resent market data).

We have done a lot but the results surely can be improved, at least one could try:

- feature engineering: polynomial features, better distances to cities (not Euclidean ones, ellipse representation of cities), average values of target for the geographically closest neighbours (requires custom estimator function for correct cross validation)
- PCA for dimensionality reduction (I have mentioned it but didn't used)
- other models (at least KNN and SVM can be tried based on data)
- more time and effort can be spent on RF and LGB parameters tuning