# Big Data and Data Science

## Big Data

“Every day, we create 2.5 quintillion bytes of data—so much that 90% of the data in the world today has been created in the last two years alone.” _IBM. 2012. “What is big data?”  Retrieved from [http://www-01.ibm.com/software/data/bigdata/](http://www-01.ibm.com/software/data/bigdata/)_ Due to the growing complexity of digital social networks and the huge quantity of data they produce daily, it’s crucial to  deal with “big data” efficiently. A number of tools and technologies (Map-Reduce, NoSQL, Hadoop, Hive, cloud computing, parallel processing, clustering, MPP, virtualization, large grid environments, and so on) have been developed to store and process big data. While big-data technologies have established the ability to collect and process large amounts of data, most organizations struggle with understanding the data and taking advantage of its value. According to an Economist report: “Extracting value from big data remains elusive for many organizations. For most companies today, data are abundant and readily available, but not well used.” _Economist. 2012. Economist Intelligence Unit Retrieved from [http://www.eiu.com/](http://www.eiu.com/)_

A central issue with “big data” is that not all of the data will be relevant or useful in solving a particular problem and will often add noise instead. However, the purpose of the present research is to better measure the system state of social networks, and this knowledge of state can help with a central issue: which data and algorithms are relevant to a particular problem? As our basic algorithm is to find tags in text, look up entropies, and sum, it is easily parallelizable and scales well to “big data.”

Central issues with big data:

* Finding the signal in the noise
* Data silos (isolated non-ingregrated data)
* Inaccurate data
* Algorithms that scale
* Echo-chamber effect (Much of big data comes from the web. Whenever the source of information for a big data analysis is itself a product of big data, opportunities for vicious cycles abound)
* Too many (spurious) correlations
* Storage
* Less structured, heterogenous
* Recognition (identifying what's what in the data)
* Discovery (efficient ways to find the specific data that can solve a particular problem)
* Semantics (effective and efficient ways to contextualize the "meaning"  of the data)
* Visualization --effective ways to analyze and visualize n-dimensional data

## Data munging

Data munging (sometimes referred to as data wrangling) is the process of transforming and mapping data from one "raw" data form into another format with the intent of making it more appropriate and valuable for a variety of downstream purposes such as analytics.

## Data cleaning

 * Save original data
 * Identify missing data
 * Identify placeholder data (e.g. 0's for NA's)
 * Identify outliers
 * Check for overall plausibility and errors (e.g., typos, unreasonable ranges)
 * Identify highly correlated variables
 * Identify variables with (nearly) no variance
 * Identify variables with strange names or values
 * Check variable classes (eg. Characters vs factors)
 * Remove/transform some variables (maybe your model does not like categorial variables)
 * Rename some variables or values (if not all data is useful)
 * Check some overall pattern (statistical/ numerical summaries)
 * Possibly center/scale variables

# Exploratory Data Analysis

In the this lesson, we look statistical and graphical techniques summarize their main characteristics of one, two variable, and multi-variable data sets. To find relationships amongst variables and to find the variables which are most interesting for a particular analysis task.

Rationale: Exploratory data analysis helps one understand the data, to form and change new theories, and decide which techniques are appropriate for analysis. After a model is finished, exploratory data analysis can look for patterns in these data that may have been missed by the original hypothesis tests. Successful exploratory analyses help the researcher modify theories and refine the analysis.

* Suggest hypotheses about the causes of observed phenomena
 * Assess assumptions on which statistical inference will be based
 * Support the selection of appropriate statistical tools and techniques
 * Provide a basis for further data collection through surveys or experiments

_Five methods that are must have_:

 * Five number summaries (mean/median, min, max, q1, q3)
 * Histograms
 * Line charts
 * Box and whisker plots
 * Pairwise scatterplots (scatterplot matrices)

 * What values do you see?
 * What distributions do you see?
 * What relationships do you see?
 * What relationships do you think might benefit the prediction problem?

* Answer the following questions for the data in each column:
    * How is the data distributed?
    * Test distribution assumptions (e.G. Normal distributions or skewed?)
    * What are the summary statistics?
    * Are there anomalies/outliers?
* Identify useful raw data & transforms (e.g. log(x))
* Identify data quality problems
* Identify outliers
* Identify subsets of interest
* Suggest functional relationships

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns

# Make plots larger
plt.rcParams['figure.figsize'] = (15, 9)
```

```python
iris = sns.load_dataset("iris")
sns.pairplot(iris, hue="species")
```

## Probability, random variables and probability distributions

* Probability
* Random variables
* Probability distributions
    * Uniform
    * Normal
    * Binomial
    * Poisson
    * Fat Tailed

## Probability

* Probability is a measure of the likelihood of a random phenomenon or chance behavior.  Probability describes the long-term proportion with which a certain outcome will occur in situations with short-term uncertainty.
* Probability is expressed in numbers between 0 and 1.  Probability = 0 means the event never happens; probability = 1 means it always happens.
* The total probability of all possible event always sums to 1.

## Sample Space

* Coin Toss ={head,tail}
* Two coins S = {HH, HT, TH, TT}
* Inspecting a part ={good,bad}
* Rolling a die S ={1,2,3,4,5,6}

## Random Variables

In probability and statistics, a random variable,  or stochastic variable is a variable whose value is subject to variations due to chance (i.e. it can take on a range of values)

* Coin Toss ={head,tail}
* Rolling a die S ={1,2,3,4,5,6}

Discrete Random Variables

* Random variables (RVs) which may take on only a countable number of distinct values
E.g. the total number of tails X you get if you flip 100 coins
* X is a RV with arity k if it can take on exactly one value out of {x1, …, xk}
E.g. the possible values that X can take on are 0, 1, 2, …, 100

Continuous Random Variables

* Probability density function (pdf) instead of probability mass function (pmf)
* A pdf is any function f(x) that describes the probability density in terms of the input variable x.

## Probability distributions

* We use probability distributions because they model data in real world.
* They allow us to calculate what to expect and therefore understand what is unusual.
* They also provide insight in to the process in which real world data may have been generated.
* Many machine learning algorithms have assumptions based on certain probability distributions.

_Cumulative distribution function_

A probability distribution Pr on the real line is determined by the probability of a scalar random variable X being in a half-open interval (-$\infty$, x], the probability distribution is completely characterized by its cumulative distribution function:

$$
 F(x) = \Pr[X \leq x] \quad \forall \quad x \in R .
$$

## Uniform Distribution

$$
X \equiv U[a,b]
$$

$$
 f(x) = \frac{1}{b-a} \quad for \quad a \lt x \lt b
$$

$$
 f(x) = 0 \quad for \quad a \leq x  \quad or  \quad \geq b
$$

$$
 F(x) = \frac{x-a}{b-a} \quad for \quad a \leq x \lt b
$$

$$
F(x) = 0 \quad for \quad x  \lt a  \quad
 F(x) = 1 \quad for \quad x  \geq b
$$

_Continuous Uniform Distribution_

In probability theory and statistics, the continuous uniform distribution or rectangular distribution is a family of symmetric probability distributions such that for each member of the family, all intervals of the same length on the distribution's support are equally probable.

- from [Uniform distribution (continuous  Wikipedia)](https://en.wikipedia.org/wiki/Uniform_distribution_(continuous))

_Discrete Uniform Distribution_

In probability theory and statistics, the discrete uniform distribution is a symmetric probability distribution whereby a finite number of values are equally likely to be observed; every one of n values has equal probability 1/n. Another way of saying "discrete uniform distribution" would be "a known, finite number of outcomes equally likely to happen".

- from [Uniform distribution (discrete)  Wikipedia)](https://en.wikipedia.org/wiki/Uniform_distribution_(discrete))

## Normal Distribution

In probability theory, the normal (or Gaussian) distribution is a very common continuous probability distribution. The normal distribution is remarkably useful because of the central limit theorem. In its most general form, under mild conditions, it states that averages of random variables independently drawn from independent distributions are normally distributed. Physical quantities that are expected to be the sum of many independent processes (such as measurement errors) often have distributions that are nearly normal.

- from [Normal Distribution - Wikipedia)](https://en.wikipedia.org/wiki/Normal_distribution)

$$
X \sim \quad N(\mu, \sigma^2)
$$

$$
 f(x) = \frac{1}{\sigma \sqrt {2\pi }} e^{-\frac{( x - \mu)^2}{2\sigma^2}} \quad
$$

Normal cumulative distribution function

_Properties of normal distribution_

- symmetrical, unimodal, and bell-shaped
- on average, the error component will equal zero, the error above and below the mean will cancel out
- Z-Score is a statistical measurement is (above/below) the mean of the data
- important characteristics about z scores:
  1. mean of z scores is 0
  2. standard deviation of a standardized variable is always 1
  3. the linear transformation does not change the _form_ of the distribution

The normal (or Gaussian) distribution was discovered in 1733 by Abraham de Moivre as an approximation to the binomial distribution when the number of trails is large.

- from [Abraham de Moivre - Wikipedia)](https://en.wikipedia.org/wiki/Abraham_de_Moivre)

The Gaussian distribution was derived in 1809 by Carl Friedrich Gauss.

- from [Carl Friedrich Gauss - Wikipedia)](https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss)

Importance lies in the Central Limit Theorem, which states that the sum of a large number of independent random variables (binomial, Poisson, etc.) will approximate a normal distribution

## Central Limit Theorem

In probability theory, the central limit theorem (CLT) states that, given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, will be approximately normally distributed, regardless of the underlying distribution. The central limit theorem has a number of variants. In its common form, the random variables must be identically distributed.

- from [Central Limit Theorem - Wikipedia)](https://en.wikipedia.org/wiki/Central_limit_theorem)

The Central Limit Theorem tells us that when the sample size is large the average $\bar{Y}$ of a random sample follows a normal distribution centered at the population average $\mu_Y$ and with standard deviation equal to the population standard deviation $\sigma_Y$, divided by the square root of the sample size $N$.

This means that if we subtract a constant from a random variable, the mean of the new random variable shifts by that constant. If $X$ is a random variable with mean $\mu$ and $a$ is a constant, the mean of $X - a$ is $\mu-a$.

This property also holds for the spread, if $X$ is a random variable with mean $\mu$ and SD $\sigma$, and $a$ is a constant, then the mean and SD of $aX$ are $a \mu$ and $\|a\| \sigma$ respectively.
This implies that if we take many samples of size $N$ then the quantity

$$
\frac{\bar{Y} - \mu}{\sigma_Y/\sqrt{N}}
$$

is approximated with a normal distribution centered at 0 and with standard deviation 1.

## The t-distribution

In probability and statistics, Student's t-distribution (or simply the t-distribution) is any member of a family of continuous probability distributions that arises when estimating the mean of a normally distributed population in situations where the sample size is small and population standard deviation is unknown. Whereas a normal distribution describes a full population, t-distributions describe samples drawn from a full population; accordingly, the t-distribution for each sample size is different, and the larger the sample, the more the distribution resembles a normal distribution.
The t-distribution plays a role in a number of widely used statistical analyses, including the Student's t-test for assessing the statistical significance of the difference between two sample means, the construction of confidence intervals for the difference between two population means, and in linear regression analysis. The Student's t-distribution also arises in the Bayesian analysis of data from a normal family.

- from [The t-distribution - Wikipedia)](https://en.wikipedia.org/wiki/Student%27s_t-distribution)

When the CLT does not apply (i.e. as the number of samples is large), there is another option that does not rely on large samples When a the original population from which a random variable, say $Y$, is sampled is normally distributed with mean 0 then we can calculate the distribution of

number of variants. In its common form, the random variables must be identically distributed.

$$
\sqrt{N} \frac{\bar{Y}}{s_Y}
$$

Normal cumulative distribution function

_Binomial Distribution_

In probability theory and statistics, the binomial distribution with parameters n and p is the discrete probability distribution of the number of successes in a sequence of n independent yes/no experiments, each of which yields success with probability p. A success/failure experiment is also called a Bernoulli experiment or Bernoulli trial; when n = 1, the binomial distribution is a Bernoulli distribution.

- from [Binomial Distribution - Wikipedia](https://en.wikipedia.org/wiki/Binomial_distribution)

Binomial Distribution

Binomial cumulative distribution function

* The data arise from a sequence of n independent trials.
* At each trial there are only two possible outcomes, conventionally called success and failure.
* The probability of success, p, is the same in each trial.
* The random variable of interest is the number of successes, X, in the n trials.
* The assumptions of independence and constant p are important. If they are invalid,  so is the binomial distribution

_Bernoulli Random Variables_

* Imagine a simple trial with only two possible outcomes
    * Success (S) with probabilty p.
    * Failure (F) with probabilty 1-p.

* Examples
    * Toss of a coin (heads or tails)
    * Gender of a newborn (male or female)

## Poisson Distribution

$X$ expresses the number of "rare" events

$$
X \quad \sim P( \lambda )\quad \lambda \gt 0
$$

$$
    P(X = x) = \frac{ \mathrm{e}^{- \lambda } \lambda^x }{x!}  \quad x=1,2,...,n
$$

_Poisson Distribution_

In probability theory and statistics, the Poisson distribution, named after French mathematician Siméon Denis Poisson, is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time and/or space if these events occur with a constant rate per time unit and independently of the time since the last event. The Poisson distribution can also be used for the number of events in other specified intervals such as distance, area or volume.

For instance, an individual keeping track of the amount of mail they receive each day may notice that they receive an average number of 4 letters per day. If receiving any particular piece of mail doesn't affect the arrival times of future pieces of mail, i.e., if pieces of mail from a wide range of sources arrive independently of one another, then a reasonable assumption is that the number of pieces of mail received per day obeys a Poisson distribution. Other examples that may follow a Poisson: the number of phone calls received by a call center per hour, the number of decay events per second from a radioactive source, or the number of taxis passing a particular street corner per hour.

The Poisson distribution gives us a probability mass for discrete natural numbers *k* given some mean value λ. Knowing that, on average, λ discrete events occur over some time period, the Poisson distribution gives us the probability of seeing exactly *k* events in that time period.

For example, if a call center gets, on average, 100 customers per day, the Poisson distribution can tell us the probability of getting exactly 150 customers today.

*k* ∈ **N** (i.e. is a natural number) because, on any particular day, you can't have a fraction of a phone call. The probability of any non-integer number of people calling in is zero. E.g., P(150.5) = 0.

λ ∈ **R** (i.e. is a real number) because, even though any *particular* day must have an integer number of people, the *mean* number of people taken over many days can be fractional (and usually is). It's why the "average" number of phone calls per day could be 3.5 even though half a phone call won't occur.

- from [Poisson Distribution - Wikipedia)](https://en.wikipedia.org/wiki/Poisson_distribution)

Poisson Distribution

Poisson cumulative distribution function

_Properties of Poisson distribution_

* The mean number of successes from n trials is µ = np
* If we substitute µ/n for p, and let n tend to infinity, the binomial distribution becomes the Poisson distribution.
* Poisson distributions are often used to describe the number of occurrences of a ‘rare’ event. For example
    * The number of storms in a season
    * The number of occasions in a season when river levels exceed a certain value
* The main assumptions are that events occur
    * at random (the occurrence of an event doesn’t change the probability of  it happening again)
    *  at a constant rate
* Poisson distributions also arise as approximations  to  binomials when n is large and p is small.
* When there is a large number of trials, but a very small probability of success, binomial calculation becomes impractical

## Deriving the Poisson Distribution from the Binomial Distribution

Let’s make this a little more formal. The binomial distribution works when we have a fixed number of events n, each with a constant probability of success p. In the Poisson Distribution, we don't know the number of trials that will happen. Instead, we only know the average number of successes per time period, the rate $\lambda$. So we know the rate of successes per day, or per minute but not the number of trials n or the probability of success p that was used to estimate to that rate.

If n is the number of trails in our time period, then np is the success rate or $\lambda$, that is, $\lambda$ = np. Solving for p, we get:

$$
p=\frac{\lambda}{n} \quad(1)
$$
Since the Binomial distribution is defined as below
$$
 P(X=k) = \binom{n}{k} p^k (1-p)^{n-k} \quad k=1,2,...,n
\quad  (2)
$$
or equivelently
$$
 P(X=k) = \frac{n!}{k!(n-k)!} p^k (1-p)^{n-k} \quad k=1,2,...,n
\quad  (3)
$$
By substituting the above p from (1) into the binomial distribution (3)
 $$
 P(X=k) = \frac{n!}{k!(n-k)!} {\frac{\lambda!}{n}}^k (1-{\frac{\lambda!}{n} })^{n-k}  \quad  (4)
 $$

For n large and p small:

$$
    P(X = k) \equiv \frac{ \mathrm{e}^{- \lambda } \lambda^k }{k!}  \quad k=1,2,...,n\quad  (5)
$$

Which is the probability mass function for the Poisson distribution.

## Fat-Tailed Distribution

In probability theory, the Fat-Tailed (or Gaussian) distribution is a very common continuous probability distribution. The Fat-Tailed distribution is remarkably useful because of the central limit theorem. In its most general form, under mild conditions, it states that averages of random variables independently drawn from independent distributions are Fat-Tailedly distributed. Physical quantities that are expected to be the sum of many independent processes (such as measurement errors) often have distributions that are nearly Fat-Tailed.

- from [Fat-Tailed Distribution - Wikipedia)](https://en.wikipedia.org/wiki/Fat-Tailed_distribution)

_Properties of Fat-Tailed distribution_

* Power law distributions:
    * for variables assuming integer values > 0
    * Prob [X=k] ~ Ck-α
    * typically 0 < alpha < 2; smaller a gives heavier tail
* For binomial, normal, and Poisson distributions the tail probabilities approach 0 exponentially fast
* What kind of phenomena does this distribution model?
* What kind of process would generate it?

## Cauchy Distribution

An example of a Fat-tailed distribution is the Cauchy distribution.

_Cauchy Distribution_

 The Cauchy distribution, named after Augustin Cauchy, is a continuous probability distribution. It is also known, especially among physicists, as the Lorentz distribution (after Hendrik Lorentz), Cauchy–Lorentz distribution, Lorentz(ian) function, or Breit–Wigner distribution. The simplest Cauchy distribution is called the standard Cauchy distribution. It is the distribution of a random variable that is the ratio of two independent standard normal variables and has the probability density function

The Cauchy distribution is often used in statistics as the canonical example of a "pathological" distribution since both its mean and its variance are undefined. (But see the section Explanation of undefined moments below.) The Cauchy distribution does not have finite moments of order greater than or equal to one; only fractional absolute moments exist.[1] The Cauchy distribution has no moment generating function.

- from [Cauchy Distribution - Wikipedia)](https://en.wikipedia.org/wiki/Cauchy_distribution)

Cauchy Distribution

Cauchy cumulative distribution function

## Unsupervised Learning

Unsupervised Learning tries to find hidden structure in unlabeled data. Unsupervised Learning has no feedback. This typically means that the data is not labled. Supervised learning has feedback/labels.

For example, if we had some e-mail that labbled some e-mail as spam and not-spam we could use a supervised learning algorthm like [Naive Bayes spam filtering](https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering) to classify any new e-mail as spam or non-spam.

However, if we didn't have labeled data (also called training data) we might still be able to cluster spam and non-spam if we could come up with some measure of similarity or distance between words in the e-mails.

*Unsupervised Learning is computational machinary to extract patterns from unlabled data*

Image courtesy of [clipartist.net](http://clipartist.net/)

[Clustering](https://en.wikipedia.org/wiki/Cluster_analysis) is probably the most common technique whose goal is to group data into similar groups based on a similarity/distance measure. Most of the focus of this module will be on clustering. We'll discuss and implement clustering in Lesson 2 of this module

#### Types of  Clustering

### Partitioning-based clustering

Partition-based clustering iterativley assigns data to groups based on a [disatnce metric](https://en.wikipedia.org/wiki/Metric_(mathematics)) or [similarity](https://en.wikipedia.org/wiki/Similarity_measure)/dissimilarity measures until the group assignments are stable (i.e. the algorithm converges.)

*K-means clustering*

Image courtesy of [clipartist.net](http://clipartist.net/)

### Hierarchical clustering

[Hierarchical clustering](https://en.wikipedia.org/wiki/Hierarchical_clustering) is a method of cluster analysis which builds a hierarchy of clusters either from the "bottom up" (agglomerative hierarchical clustering) by combiinimg the two "closet" or "most similar" points into a cluster then the next two closet, until all points are  merged as one moves up the hierarcy.

Conversely the data is split "top down" (divisive hierarchical clustering) so that there is the most seperation between the split groups and further splits are performed recursively untl there is a single data point in each group.

*Hierarchical clustering*

Image courtesy of [clipartist.net](http://clipartist.net/)

### Density-based clustering

ensity-based clustering is based on probabilty distribution models. Clusters are defined as a set of distibutions then assigning points to the distibutions that they are most likely to belong. The most prominent method is known as Gaussian [mixture models](https://en.wikipedia.org/wiki/Mixture_model) that represent the clusters as multivariate normal distributions.

$$ p(\boldsymbol{\theta}) = \sum_{i=1}^K\phi_i \mathcal{N}(\boldsymbol{\mu_i,\sigma_i}) $$

*Gaussian mixture model*

Image courtesy of [clipartist.net](http://clipartist.net/)

## Expectation-maximization

The [Expectation-maximization algorithm](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm) is an iterative method for finding [maximum likelihood](https://en.wikipedia.org/wiki/Maximum_likelihood) or maximum a posteriori (MAP) estimates of parameters in statistical models, where the model depends on unobserved latent variables. We'll discuss and implement association rule learning in Lesson 4 of this module.

Given a statistica which generates a set $\mathbf{X}$ of observed data, a set of unobserved latent data or missing values $\mathbf{Z}$, and a vector of unknown parameters $\boldsymbol\theta$, along with a likelihood function $L(\boldsymbol\theta; \mathbf{X}$, $\mathbf{Z}) = p(\mathbf{X}$, $\mathbf{Z}|\boldsymbol\theta)$, the maximum likelihood estimate (MLE) of the unknown parameters is determined by the marginal likelihood of the observed data.

$$ L(\boldsymbol\theta; \mathbf{X}) = p(\mathbf{X}|\boldsymbol\theta) = \sum_{\mathbf{Z}} p(\mathbf{X},\mathbf{Z}|\boldsymbol\theta) $$

However, this quantity is often intractable (e.g. if $\mathbf{Z}$ is a sequence of events, so that the number of values grows exponentially with the sequence length, making the exact calculation of the sum extremely difficult).

The EM algorithm seeks to find the MLE of the marginal likelihood by iteratively applying the following two steps:

*Expectation step (E step):* Calculate the expected value of the log likelihood function, with respect to the conditional distribution of $\mathbf{Z}$ given $\mathbf{X}$ under the current estimate of the parameters
 $$
\boldsymbol\theta^{(t)}:
Q(\boldsymbol\theta|\boldsymbol\theta^{(t)}) = \operatorname{E}_{\mathbf{Z}|\mathbf{X},\boldsymbol\theta^{(t)}}\left[ \log L (\boldsymbol\theta;\mathbf{X},\mathbf{Z})  \right]  $$

*Maximization step (M step):* Find the parameter that maximizes this quantity:
 $$
\boldsymbol\theta^{(t+1)} = \underset{\boldsymbol\theta}{\operatorname{arg\,max}} \ Q(\boldsymbol\theta|\boldsymbol\theta^{(t)}) $$

## Linear Discriminant Analysis (LDA)

[Linear Discriminant Analysis (LDA)](https://en.wikipedia.org/wiki/Linear_discriminant_analysis) which is related to blind signal separation techniques, such as those we've already studied in the dimensionality reduction module like: Principal component analysis, Independent component analysis,Non-negative matrix factorization, and  Singular value decomposition. LDA explicitly attempts to model the difference between the classes of data. That is,  like regression analysis, LDA attempts to express one response variable as a linear combination of other features or measurements. So while the math is similar to PCA and SVD, the intent is similar to regression (especially logistic regression) in that it creates a model. We'll discuss and implement association rule learning in Lesson 4 of this module.

LDA is often based upon s Fisher's linear discriminant. Fisher defined the separation between these two distributions to be the ratio of the variance between the classes to the variance within the classes:

$$ S=\frac{\sigma_{\text{between}}^2}{\sigma_{\text{within}}^2}= \frac{(\vec w \cdot \vec \mu_1 - \vec w \cdot \vec \mu_0)^2}{\vec w^T \sigma_1 \vec w + \vec w^T \sigma_0 \vec w} = \frac{(\vec w \cdot (\vec \mu_1 - \vec \mu_0))^2}{\vec w^T (\sigma_0+\sigma_1) \vec w} $$

This measure is, in some sense, a measure of the signal-to-noise ratio for the class labelling. It can be shown that the maximum separation occurs when

$$  \vec{w} \propto (\sigma_0+\sigma_1)^{-1}(\vec{\mu}_1 - \vec{\mu}_0) $$
When the assumptions of LDA are satisfied, the above equation is equivalent to LDA.

In essence, LDA picks a new basis that gives:

To do this, LDA uses eigenvectors based on between-class and  within-class covariance matrices.

$$ max\frac{(\mu_1 - \mu_2)^2}{\sigma_1^2 + \sigma_2^2}  $$

Image courtesy of [clipartist.net](http://clipartist.net/)

## Association rule learning

[Association rule learning](https://en.wikipedia.org/wiki/Association_rule_learning) is an Unsupervised alogorthm interesting relations between variables in a data set. We'll discuss and implement association rule learning in Lesson 5 of this module. Association rule learning aims to discover interesting correlation or other relationships in large databases. It finds a rule of the form:

$$ if\quad A \quad and \quad B \quad then \quad C \quad and \quad D $$

*Association rule Set Enumeration Tree*

## Artificial neural networks (ANN)

[Artificial neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network) are a  are a family of statistical learning models inspired by biological neural networks. Some neural network models like
[Self-organizing map's](https://en.wikipedia.org/wiki/Self-organizing_map),[adaptive resonance theory (ART)](https://en.wikipedia.org/wiki/Adaptive_resonance_theory), [Restricted Boltzmann machines](https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine) and [Hopfield network's](https://en.wikipedia.org/wiki/Hopfield_network). We'll discuss and implement various neural statistical learning models in a module entirely devoted to neural networks. We'll simply note here that some statistical neural learning models are unsupervised and put off discussing the details until the neural network module.

An illustration of the training of a self-organizing map

Art ["Somtraining"](https://commons.wikimedia.org/wiki/File:Somtraining.svg##/media/File:Somtraining.svg) by [Mcld](http://www.mcld.co.uk/). Licensed under [CC BY-SA 3.0](http://creativecommons.org/licenses/by-sa/3.0)

*An illustration of a stable Hopfield network*

Image courtesy of [clipartist.net](http://clipartist.net/)

## Big Data and Unsupervised Learning

“Every day, we create 2.5 quintillion bytes of data—so much that 90% of the data in the world today has been created in the last two years alone.” $[1]$ Due to the growing complexity of digital social networks and the huge quantity of data they produce daily, it’s important that we deal with “big-data” efficiently. A number of tools and technologies (Map-Reduce, NoSQL, Hadoop, Hive, cloud computing, parallel processing, clustering, MPP, virtualization, large grid environments, and so on) have been developed to store and process big data. While big-data technologies have established the ability to collect and process large amounts of data, most organizations struggle with understanding the data and taking advantage of its value. According to an Economist report: “Extracting value from big data remains elusive for many organizations. For most companies today, data are abundant and readily available, but not well used.”$[2,3]$

A central issue with “big data” is that not all of the data will labled. As such, there is an expectation that unsupervised learning and research in unsupervised learning will take on an even greater importance in machine learning.

Image courtesy of [clipartist.net](http://clipartist.net/)

## Clustering

What is [Clustering](https://en.wikipedia.org/wiki/Cluster_analysis)?
Clustering is grouping like with like such that:

* Similar objects are close to one another within the same cluster.
* Dissimilar to the objects in other clusters.

*cluster analysis*

# Distance & Similarity Measures

There are two primary approaches to measure the "closeness" of data, distance and similarity. Distance measures are based in the notion of a [metric space](https://en.wikipedia.org/wiki/Metric_space).

[Similarity measures](https://en.wikipedia.org/wiki/Similarity_measure) s a real-valued function that quantifies the similarity between two objects. While no single definition of a similarity measure exists, they don't necessarily have the constraint of being a metric space.  For example, [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) is a measure of the "closeness" of two vectors that is not a metric space.

## Metric Spaces

A *metric space* is an *ordered pair* $(M,d)$ where $M$ is a set and $d$ is a distance (i.e.metric) on $M$, i.e., a function:

$d \colon M \times M \rightarrow \mathbb{R}$

such that for any $x, y, z \in M$, the following holds:

* $d(x,y) \ge 0 \quad (non-negative),$
* $d(x,y) = 0, \iff x = y \quad (identity \quad of \quad indiscernibles),$
* $d(x,y) = d(y,x), \quad (symmetry).$
* $d(x,z) \le d(x,y) + d(y,z) \quad  (triangle \quad inequality)$

### Examples of Metric Spaces

Examples of Metric Spaces are [Chebyshev distance](https://en.wikipedia.org/wiki/Chebyshev_distance), [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance), [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance), [Minkowski distance](https://en.wikipedia.org/wiki/Minkowski_distance) and many others.

#### Euclidean distance

Euclidean distance is the most common metric for measuring the distance between two vectors. The is the stadard Cartesian coordinates. That is, if $p = (p_1, p_2,..., p_n)$ and $q = (q_1, q_2,..., q_n)$ are two points in Euclidean n-space, then the distance (d) from p to q, or from q to p is given by the Pythagorean formula:

$$\begin{align}\mathrm{d}(\mathbf{p},\mathbf{q}) = \mathrm{d}(\mathbf{q},\mathbf{p}) & = \sqrt{(q_1-p_1)^2 + (q_2-p_2)^2 + \cdots + (q_n-p_n)^2} \\[8pt]
& = \sqrt{\sum_{i=1}^n (q_i-p_i)^2}.\end{align}$$

## Similarity Measures

Similarity (or dimilarity measures) measure closeness without the constraints and benefits of being a formal metric space.

### Cosine similarity

The most common form of the similarity measure is the vector inner product (or [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity))
Given vectors A and B, the vector inner product can be defined using the [Euclidean dot product](https://en.wikipedia.org/wiki/Euclidean_vector#Dot_product) formula:

$$\mathbf{a}\cdot\mathbf{b}
=\left\|\mathbf{a}\right\|\left\|\mathbf{b}\right\|\cos\theta$$

This similarity measure can also be ranged normalized. Alternately, we can normalize this measure by dividing each vector component by the magnitude of the vector.

### Pearson correlation

Correlation based similarity is usually the [Pearson correlation](https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient). Pearson product-moment correlation coefficient commonly represented by the Greek letter $\rho$ (rho) and is defined as:

$$ \rho_{X,Y}= \frac{\operatorname{cov}(X,Y)}{\sigma_X \sigma_Y} $$

where $\operatorname{cov}$ is the [covariance](https://en.wikipedia.org/wiki/Covariance) and  $\sigma_X$ is the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of $X$.

The formula for $\rho$can be expressed in terms of mean and expectation.

$$\operatorname{cov}(X,Y) = E[(X-\mu_X)(Y-\mu_Y)]$$

# Clustering Data Structures

* Data matrix

$$
M=
  \begin{bmatrix}
    x_{1,1} & x_{1,2} & x_{1,3} & ...  & x_{1,n} \\
    x_{2,1} & x_{2,2} & x_{2,3} & ...  & x_{2,n} \\
    x_{3,1} & x_{3,2} & x_{3,3} & ...  & x_{3,n} \\
    .. & .. & .. & ...  & .. \\
    x_{n,1} & x_{n,2} & x_{n,3} & ...  & x_{n,n} \\
  \end{bmatrix}
$$

Distance matrix

$$
M=
  \begin{bmatrix}
    x_{1,1} &   &  &   &  \\
    x_{2,1} & 0  &  &   &  \\
    x_{3,1} & x_{3,2} & 0 &   &  \\
    .. & .. & .. & ...  & .. \\
    x_{n,1} & x_{n,2} & x_{n,3} & ...  & 0 \\
  \end{bmatrix}
$$

* Dissimilarity/Similarity matrix

The dissimilarity/similarity matrix is calculated by iterating over each element and calculating its dissimilarity/similarity to every other element. Let A be a Dissimilarity Matrix of size $NxN$, and $B$ a set of $N$ elements. $A_{i,j}$ is the dissimilarity/similarity between elements $B_i$ and $B_j$.

```

   for i = 0 to N do
       for j = 0 to N do
           Aij = Dissimilarity(Bi,Bj) // or Similarity(Bi,Bj)
       end-for
   end-for

```

where the dissimilarity/similarity matrix is usually defined as follows:

$$
M=
  \begin{bmatrix}
    0 &   &  &   &  \\
   d(2,1)  & 0  &  &   &  \\
   d(3,1)  & d(3,2) & 0 &   &  \\
    .. & .. & .. & ...  & .. \\
    d(n,1)  & d(n,2) & d(n,3) & ...  & 0 \\
  \end{bmatrix}
$$

# Types of  Clustering

[Clustering](https://en.wikipedia.org/wiki/Cluster_analysis) (e.g., k-means, mixture models, hierarchical clustering). Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). There are various types of cluster analysis.

* Partitioning-based clustering (K-means and its variants)
* Hierarchical clustering
* Density-based clustering

## Partitioning-based clustering (K-means and its variants)

Partitioning algorithms: Construct various partitions and then evaluate them by some criterion
Hierarchy algorithms: Create a hierarchical decomposition of the set of data (or objects) using some criterion
Density-based: based on connectivity and density functions
Grid-based: based on a multiple-level granularity structure
Model-based: A model is hypothesized for each of the clusters and the idea is to find the best fit of that model to each other

Partitioning method: Construct a partition of n documents into a set of K clusters
Given: a set of documents and the number K
Find: a partition of K clusters that optimizes the chosen partitioning criterion
Globally optimal
Intractable for many objective functions
Ergo, exhaustively enumerate all partitions
Effective heuristic methods: K-means and K-medoids algorithms

### K-means

The term "k-means" was first used by James MacQueen in 1967,[1] though the idea goes back to Hugo Steinhaus in 1957. Given a desired $k$ clusters and n data points $X = {x_1, x_2, …, x_n}$:

Initialize centroids $\mu_1, \mu_1, ... \mu_k \quad \in \quad  \mathbb{R}^n$  (usually randomly)

while (not coverged):

Step A (Assignment step):

  Find the closest cluster to every point in $X = {x_1, x_2, …, x_n}$

  That is,
$$\underset{\mathbf{X}} {\operatorname{arg\,min}}  \sum_{i=1}^{k} \sum_{\mathbf x \in X_i} \left\| \mathbf x - \boldsymbol\mu_i \right\|^2$$

where $\mu_i$ is the mean of points in $X_i$.

$$
S_i^{(t)} = \big \{ x_p : \big \| x_p - m^{(t)}_i \big \|^2 \le \big \| x_p - m^{(t)}_j \big \|^2 \ \forall j, 1 \le j \le k \big\}
$$

Step B (Update step):

  Calculate the new means to be the centroids of the observations in the new clusters.

$$
m^{(t+1)}_i = \frac{1}{|X^{(t)}_i|} \sum_{x_j \in X^{(t)}_i} x_j
$$

The [centroid](https://en.wikipedia.org/wiki/Centroid) of a finite set of {k} points

$$
\mathbf{x}_1,\mathbf{x}_2,\ldots,\mathbf{x}_k in \mathbb{R}^n \quad is\quad
\mathbf{C} = \frac{\mathbf{x}_1+\mathbf{x}_2+\cdots+\mathbf{x}_k}{k}
$$

## Supervised Learning

[Supervised learning](https://en.wikipedia.org/wiki/Supervised_learning) is the machine learning task of inferring a function from labeled training data.[1] The training data consist of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal). A supervised learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples. An optimal scenario will allow for the algorithm to correctly determine the class labels for unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way.

## Bias–variance tradeoff

The [Bias–variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) is the problem of simultaneously minimizing two sources of error that prevent supervised learning algorithms from generalizing beyond their training set:

The bias is error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).

The variance is error from sensitivity to small fluctuations in the training set. High variance can cause an algorithm to model the random noise in the training data, rather than the intended outputs (overfitting).

The bias–variance decomposition is a way of analyzing a learning algorithm's expected generalization error with respect to a particular problem as a sum of three terms, the bias, variance, and a quantity called the irreducible error, resulting from noise in the problem itself.
This tradeoff applies to all forms of supervised learning: classification, regression (function fitting), and structured output learning. It has also been invoked to explain the effectiveness of heuristics in human learning.

This is the essence of the **bias-variance tradeoff**: You are seeking a model that appropriately balances bias and variance, and thus will generalize to new data (known as "out-of-sample" data).

### Overfitting

**What is overfitting?**

- Building a model that matches the training data "too closely"
- Learning from the noise in the data, rather than just the signal

**How does overfitting occur?**

- Evaluating a model by testing it on the same data that was used to train it
- Creating a model that is "too complex"

**What is the impact of overfitting?**

- Model will do well on the training data, but won't generalize to out-of-sample data
- Model will have low bias, but high variance

## Linear Models and Regression

Linear regression predicts the response variable $y$ assuming it has a linear relationship with predictor variable(s) $x$ or $x_1, x_2, ,,, x_n$.

$$y = \beta_0 + \beta_1 x + \varepsilon .$$

*_Simple regression_* use only one predictor variable $x$. *_Mulitple regression_* uses a set of predictor variables $x_1, x_2, ,,, x_n$.

The *response variable* $y$ is also called the regressand, forecast, dependent or explained variable. The *predictor variable* $x$ is also called the regressor, independent or explanatory variable.

The parameters $\beta_0$ and $\beta_1$ determine the intercept and the slope of the line respectively. The intercept $\beta_0$ represents the predicted value of $y$ when $x=0$. The slope $\beta_1$ represents the predicted increase in $Y$ resulting from a one unit increase in $x$.

Note that the regression equation is just our famliar equation for a line with an error term.

The equation for a line:
$$ Y = bX + a $$

$$y = \beta_0 + \beta_1 x $$

The equation for a line with an error term:

$$ Y = bX + a + \varepsilon $$

$$y = \beta_0 + \beta_1 x + \varepsilon .$$

- $b$ = $\beta_1$ = slope
- $a$ = $\beta_0$ = $Y$ intercept
- $\varepsilon$ = error term

We can think of each observation $y_i$ consisting of the systematic or explained part of the model, $\beta_0+\beta_1x_i$, and the random *error*, $\varepsilon_i$.

_Zero Slope_

Note that when  $\beta_1 = 0$ then response does not change as the predictor changes.

## The error $\varepsilon_i$

The error term is a catch-all for anything that may affect $y_i$ other than $x_i$. We assume that these errors:

* have mean zero; otherwise the forecasts will be systematically biased.
* statistical independence of the errors (in particular, no correlation between consecutive errors in the case of time series data).
* homoscedasticity (constant variance) of the errors.
* normality of the error distribution.

If any of these assumptions is violated then the robustness of the model to be taken with a grain of salt.

## Least squares estimation

In a linear model, the values of $\beta_0$ and $\beta_1$. These need to be estimated from the data. We call this *fitting a model*.

The least squares method iis the most common way of estimating $\beta_0$ and $\beta_1$ by minimizing the sum of the squared errors. The values of $\beta_0$ and $\beta_1$ are chosen so that that minimize

$$\sum_{i=1}^N \varepsilon_i^2 = \sum_{i=1}^N (y_i - \beta_0 - \beta_1x_i)^2. $$

Using mathematical calculus, it can be shown that the resulting **least squares estimators** are

$$\hat{\beta}_1=\frac{ \sum_{i=1}^{N}(y_i-\bar{y})(x_i-\bar{x})}{\sum_{i=1}^{N}(x_i-\bar{x})^2} $$

and

$$\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}, $$

where $\bar{x}$ is the average of the $x$ observations and $\bar{y}$ is the average of the $y$ observations. The estimated line is known as the *regression line*.

## Fitted values and residuals

The response values of $y$ obtained from the observed $x$ values are
called *fitted values*: $\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_i$, for
$i=1,\dots,N$. Each $\hat{y}_i$ is the point on the regression
line corresponding to $x_i$.

The difference between the observed $y$ values and the corresponding fitted values are the *residuals*:

$$e_i = y_i - \hat{y}_i = y_i -\hat{\beta}_0-\hat{\beta}_1x_i. $$

The residuals have some useful properties including the following two:

$$\sum_{i=1}^{N}{e_i}=0 \quad\text{and}\quad \sum_{i=1}^{N}{x_ie_i}=0. $$

Residuals are the errors that we cannot predict.Residuals are highly useful for studying whether a given regression model is an appropriate statistical technique for analyzing the relationship.

## Regression and correlation

The correlation coefficient $r$ measures the strength and the direction of the linear relationship between the two variables. The stronger the linear relationship, the closer the observed data points will cluster around a straight line.

The _Pearson product-moment correlation coefficient_ is the most widely used of all correlation coefficients. In statistics, the Pearson product-moment correlation coefficient (/ˈpɪərsɨn/) (sometimes referred to as the PPMCC or PCC or Pearson's r) is a measure of the linear correlation (dependence) between two variables X and Y, giving a value between +1 and −1 inclusive, where 1 is total positive correlation, 0 is no correlation, and −1 is total negative correlation. It is widely used in the sciences as a measure of the degree of linear dependence between two variables. It was developed by Karl Pearson from a related idea introduced by Francis Galton in the 1880s. Early work on the distribution of the sample correlation coefficient was carried out by Anil Kumar Gain and R. A. Fisher from the University of Cambridge.

from [Pearson product-moment correlation coefficient](https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient)

Examples of scatter diagrams with different values of correlation coefficient (ρ)

The value of r is such that -1 < r < +1.

Strong positive correlation r is close to +1. Strong negative correlation r is close to -1. No correlation r is close to 0.

The advantage of a regression model over correlation is that it asserts a predictive relationship between the two variables ($x$ predicts $y$) and quantifies this in a useful way for forecasting.

## The t-statistic and the standard error

The t-statistic is the coefficient divided by its standard error. For example, a $\beta_1$ of 38.2 divided by a standard error of 3.4 would give a t value of 11.2.

The standard error is an estimate of the standard deviation of the coefficient. It can be thought of as the spread between fitted and actual values.

$$s_e=\sqrt{\frac{1}{N-2}\sum_{i=1}^{N}{e_i^2}}.$$

If a coefficient is large compared to its standard error, then we can reject the hypothesis that $\beta_1$ = 0. Intuitively, we can think of this if the slope is not small and there is a not much spread between fitted and actual values then we can be confident that the true slope $\hat{\beta}_1$ is not 0.

A t-statistic (t value) of greater than 2 in magnitude, corresponds to p-values less than 0.05.

The p-value is a function of the observed sample results (a statistic) that is used for testing a statistical hypothesis. Before the test is performed, a threshold value is chosen, called the significance level of the test, traditionally 5% or 1% and denoted as $\alpha$.

If the p-value is equal to or smaller than the significance level ($\alpha$), it suggests that the observed data are inconsistent with the assumption that the null hypothesis is true and thus that hypothesis must be rejected (but this does not automatically mean the alternative hypothesis can be accepted as true). When the p-value is calculated correctly, such a test is guaranteed to control the Type I error rate to be no greater than $\alpha$.

from [P-value](https://en.wikipedia.org/wiki/P-value)

## R-squared $R^2$

[R-squared](https://en.wikipedia.org/wiki/Coefficient_of_determination)   (coefficient of determination) is a statistical measure of how close the data are to the fitted regression line. the coefficient of determination, denoted $R^2$ or $r^2$ and pronounced "R squared", is a number that indicates the proportion of the variance in the dependent variable that is predictable from the independent variable(s).

R-squared = Explained variation / Total variation

R-squared is always between 0 and 100%:

* 0% (or 0) indicates that the model explains none of the variability of the response data around its mean.
* 100% (or 1) indicates that the model explains all the variability of the response data around its mean.

The higher the R-squared, the better the model fits your data.

The better the linear regression fits the data in comparison to the simple average (on the left graph), the closer the value of $R^2$ is to 1. The areas of the blue squares represent the squared residuals with respect to the linear regression. The areas of the red squares represent the squared residuals with respect to the average value.]]

A data set has $n$ values marked $y_1$,...,$y_n$ (collectively known as $y_i$ or as a vector $y = [y_1,..., y_n]^T$), each associated with a predicted (or modeled) value $\hat{y}_{1},...,\hat{y}_{n}$.

Define the residuals as as $e_i = y_i − \hat{y}_{i}$ (forming a vector $e$).

If $\bar{y}$ is the mean of the observed data:

$$\bar{y}=\frac{1}{n}\sum_{i=1}^n y_i $$

then the variability of the data set can be measured using three sums of square formulas:

* The total sum of squares (proportional to the variance of the data):

 $$SS_\text{tot}=\sum_i (y_i-\bar{y})^2,$$

* The regression sum of squares, also called the explained sum of squares:

$$SS_\text{reg}=\sum_i (\hat{y}_{i} -\bar{y})^2,$$

* The sum of squares of residuals, also called the residual sum of squares:

$$SS_\text{res}=\sum_i (y_i - \hat{y}_{i})^2=\sum_i e_i^2\,$$

The most general definition of the coefficient of determination is

$$R^2 \equiv 1 - {SS_{\rm res}\over SS_{\rm tot}}.\,$$

$R^2$ can be thought of as *the proportion of variation in the forecast variable that is accounted for (or explained) by the regression model*.

In the definition of $R^2$, $0 \geq R^2 \geq 1$ as is similar to he value of $r^2$ (the square of the pearson correlation between $f(x)$ and $x$.

### Adjusted $R^2$

The use of an adjusted $R^2$ (also called  $$\bar R^2 $$, pronounced "R bar squared" or another is  $$R^2_{\text{adj}} $$) is an attempt to take account of the phenomenon of the $R^2$   automatically and spuriously increasing when extra explanatory variables are added to the model. It is a modification due to Theil of $R^2$   that adjusts for the number of explanatory variables in a model relative to the number of data points. The adjusted $R^2$   can be negative, and its value will always be less than or equal to that of $R^2$  . Unlike $R^2$  , the adjusted $R^2$   increases only when the increase in $R^2$ (due to the inclusion of a new explanatory variable) is more than one would expect to see by chance. If a set of explanatory variables with a predetermined hierarchy of importance are introduced into a regression one at a time, with the adjusted $R^2$   computed each time, the level at which adjusted $R^2$   reaches a maximum, and decreases afterward, would be the regression with the ideal combination of having the best fit without excess/unnecessary terms. The adjusted $R^2$   is defined as

$$\bar R^2 = {1-(1-R^2){n-1 \over n-p-1}} = {R^2-(1-R^2){p \over n-p-1}} $$

where ''p'' is the total number of explanatory variables in the model (not including the constant term), and ''n'' is the sample size.

Adjusted $R^2$   can also be written as

$$\bar R^2 = {1-{SS_\text{res}/\text{df}_e \over SS_\text{tot}/\text{df}_t}} $$

where $df_{t}$ is the degrees of freedom ''n''– 1 of the estimate of the population variance of the dependent variable, and $df_{e}$ is the degrees of freedom ''n'' – ''p'' – 1 of the estimate of the underlying population error variance.

The principle behind the adjusted $R^2$   statistic can be seen by rewriting the ordinary $R^2$   as

$$R^{2} = {1-{\textit{VAR}_\text{res} \over \textit{VAR}_\text{tot}}} $$

where  $$\text{VAR}_\text{res} = SS_\text{res}/n $$ and  $$\text{VAR}_\text{tot} = SS_\text{tot}/n $$ are the sample variances of the estimated residuals and the dependent variable respectively, which can be seen as biased estimates of the population variances of the errors and of the dependent variable. These estimates are replaced by statistically unbiased versions:

$$\text{VAR}_\text{res} = SS_\text{res}/(n-p-1) $$ and  $$\text{VAR}_\text{tot} = SS_\text{tot}/(n-1) $$.

Adjusted $R^2$   does not have the same interpretation as $R^2$  —while $R^2$   is a measure of fit, adjusted $R^2$   is instead a comparative measure of suitability of alternative nested sets of explanators.  As such, care must be taken in interpreting and reporting this statistic. Adjusted $R^2$   is particularly useful in the feature selection stage of model building.

## Dimensionality reduction

Dimensionality reduction is about converting data of high dimensionality into data of lower dimensionality while keeping most of the information in the data. This allows us to work on larger datasets and identify the data’s most relevant features. Anomaly detection (or outlier detection) is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset.  In this first lesson, we study the theory of dimensionality reduction and introduce essential concepts and jargon, such as eigenvalues, eigenvectors, linear independence, span, vector, scalar, basis of a subspace and linear combination.

# Dimensionality Reduction

In machine learning and statistics, [dimensionality reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction) or dimension reduction is the process of reducing the number of random variables under consideration, and can be divided into feature selection and feature extraction.

*Feature selection*

Feature selection approaches try to find a subset of the original variables (also called features or attributes). In essence, either using domain knowledge and statisitcal tests to prune away some of the original variables

*Feature extraction*

Feature extraction transforms the data in the high-dimensional space to a space of fewer dimensions.

# Factor Analysis

Factor analysis is a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors.  The observed variables are modelled as linear combinations of the potential factors, plus "error" terms.

Factor Analysis has was first developed in 1901 by [Karl Pearson](https://en.wikipedia.org/wiki/Karl_Pearson). Pearson posed a model having one factor that was common across his data:

$$
Y_ij = \alpha_i W_1 + \varepsilon_{ij}
$$

Or a general multi-factor factor:

$$
Y_{ij} = \sum_{k=1}^K \alpha_{i,k} W_{j,k} + \varepsilon_{ij}
$$

We can use various techniques to estimate $\mathbf{W}_1,\dots,\mathbf{W}_K$. Choosing $k$ (the number of factors) is a challenge that we'll discuss in further sections.

To illustrate the idea of feature extraction using factors consider the problem of reducing 2D data to 1D.

# Linear Algebra

So how do we find a good basis to project some data?

## Some Linear Algebra Jargon

### Real coordinate spaces

$$\mathbb{R}^2 , \quad \mathbb{R}^3, ... , \quad \mathbb{R}^N$$

### What is a vector?

* A vector is a quantity having direction as well as magnitude.
* An element of the real coordinate space $\mathbb{R}^N$

### Systems of Linear Equations

Linear algebra are used to solve systems of linear equations such as this:

$$
\begin{align*}
a + b + c &= 5\\
3a - 2b + c &= 3\\
2a + b  - c &= 1
\end{align*}
$$

We can rewrite and solve this system using matrix algebra notation:

$$
\,
\begin{pmatrix}
1&1&1\\
3&-2&1\\
2&1&-1
\end{pmatrix}
\begin{pmatrix}
a\\
b\\
c
\end{pmatrix} =
\begin{pmatrix}
5\\
3\\
1
\end{pmatrix}
\implies
\begin{pmatrix}
a\\
b\\
c
\end{pmatrix} =
\begin{pmatrix}
1&1&1\\
3&-2&1\\
2&1&-1
\end{pmatrix}^{-1}
\begin{pmatrix}
5\\
3\\
1
\end{pmatrix}
$$

### Multiplying by Vector or Matrix by a Scalar

scalar multiplication of a real Euclidean vector by a positive real number multiplies the magnitude of the vector without changing its direction.

Multiplying by a negative value changes its direction.

If $a$ is scalar and $\mathbf{X}$ is a matrix then:

$$
a \mathbf{X} =
\begin{pmatrix}
a x_{1,1} & \dots & a x_{1,p}\\
& \vdots & \\
a x_{N,1} & \dots & a  x_{N,p}
\end{pmatrix}
$$

Properties of Scalar Multiplication

Scalar multiplication obeys the following rules:
* Additivity in the scalar: $(c + d)\vec{v} = c\vec{v} + d\vec{v};$
* Additivity in the vector: $c(\vec{v} + \vec{w}) = c\vec{v} + c\vec{w};$
* Compatibility of product of scalars with scalar multiplication: $(cd)\vec{v} = c(d\vec{v});$
* Multiplying by 1 does not change a vector: $1\vec{v} = \vec{v};$
* Multiplying by 0 gives the zero vector: $0\vec{v} = 0;$
* Multiplying by −1 gives the additive inverse: $(−1)\vec{v} = −\vec{v}.$

### The Matrix Transpose

The matrix transpose is an operation that changes columns to rows. We use either a $\top$ to denote transpose.

$$
\mathbf{X} = \begin{pmatrix}
x_{1,1}&\dots & x_{1,p} \\
x_{2,1}&\dots & x_{2,p} \\
& \vdots & \\
x_{N,1}&\dots & x_{N,p}
\end{pmatrix} \implies
\mathbf{X}^\top = \begin{pmatrix}
x_{1,1}&\dots & x_{p,1} \\
x_{1,2}&\dots & x_{p,2} \\
& \vdots & \\
x_{1,N}&\dots & x_{p,N}
\end{pmatrix}
$$

### Matrix multiplication

[matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication) is an operation that takes a pair of matrices, and produces another matrix. If A is an n × m matrix and B is an m × p matrix, their matrix product AB is an n × p matrix. The number rows of the first matrix must match the columns of the second.

$$
\begin{align*}
a + b + c &=5\\
3a - 2b + c &= 3\\
2a + b  - c &= 1
\end{align*}
$$

The idea is to multiply the rows of the first matrix by the columns of the second.

$$
\,
\begin{pmatrix}
1&1&1\\
3&-2&1\\
2&1&-1
\end{pmatrix}
\begin{pmatrix}
a\\
b\\
c
\end{pmatrix}=
\begin{pmatrix}
a + b + c \\
3a - 2b + c \\
2a + b  - c
\end{pmatrix}
$$

#### Adding vectors

### Unit vector

A unit vector in a normed vector space is a vector of length 1. A unit vector is often denoted by a lowercase letter with a "hat": i-hat (pronounced "i-hat"). The normalized vector or versor û of a non-zero vector u is the unit vector in the direction of u, i.e.,

$$\mathbf{\hat{u}} = \frac{\mathbf{u}}{\|\mathbf{u}\|}$$

u-hat equals the vector u divided by its length where ||u|| is the norm (or length) of u. The term normalized vector is sometimes used as a synonym for unit vector.

from [Unit vector - Wikipedia](https://en.wikipedia.org/wiki/Unit_vector)

### Linear combinations

We can represent any vector with a linear combination of other vectors. If $\vec{V}$ are vectors $v_1,...,v_n$ and A are scalars $a_1,...,a_n$ then the linear combination of those vectors with those scalars as coefficients is

$$ a_1 \vec{v}_{1} \quad + \quad a_2 \vec{v}_2  \quad + \quad  a_3 \vec{v}_3  \quad + \quad  \cdots  \quad + \quad  a_n \vec{v}_n. $$

For example, consider the vectors $\vec{v}_1 = (1,0,0),  \quad \vec{v}_2 = (0,1,0)  \quad and  \quad \vec{v}_3 = (0,0,1)$. Then any vector in $\mathbb{R}^3$ is a linear combination of $\vec{v}_1,  \quad \vec{v}_2  \quad and  \quad \vec{v}_3$.
To see that this is so, take an arbitrary vector (a1,a2,a3) in $\mathbb{R}^3,$, and write:

$$ ( a_1 , a_2 , a_3) = ( a_1 ,0,0) + (0, a_2 ,0) + (0,0, a_3)   =  a_1 (1,0,0) + a_2 (0,1,0) + a_3 (0,0,1) =  a_1 \vec{v}_1 +  a_2 \vec{v}_2 +  a_3 \vec{v}_3.   $$

### Linear Spans and Basis

The span of S may be defined as the set of all finite linear combinations of elements of S,

$$ \operatorname{span}(S) =  \left \{ {0+\sum_{i=1}^k \lambda_i v_i \Big| k \in \mathbb{N}, v_i  \in S, \lambda _i  \in \mathbf{K}} \right \} $$

A set of vectors in a vector space V is called a [basis](https://en.wikipedia.org/wiki/Basis_(linear_algebra)), or a set of basis vectors, if the vectors are linearly independent and every vector in the vector space is a linear combination of this set. In more general terms, a basis is a linearly independent spanning set.

For example, the real vector space $\mathbb{R}^3$ has {(5,0,0), (0,3,0), (0,0,9)} is a spanning set. This particular spanning set is also a basis.

The set {(1,0,0), (0,1,0), (1,3,0)} is not a spanning set of $\mathbb{R}^3$ as a vector like (1,3,1) cannot be created from this spanning set.

### Linear subspaces

A [linear subspace](https://en.wikipedia.org/wiki/Linear_subspace) (or vector subspace) is a vector space that is a subset of some other (higher-dimension) vector space. For example, $\mathbb{R}^2$ is a subspace of $\mathbb{R}^3$.

Let V be a vector space over the field K, and let W be a subset of V. Then W is a subspace if and only if W satisfies the following three conditions:

*  The zero vector, 0, is in W.
*  If u and v are elements of W, then the sum u + v is an element of W;
*  If u is an element of W and c is a scalar from K, then the product cu is an element of W;

### Metric Spaces

A metric space is an ordered pair (M,d) where M is a set and d is a metric on M, i.e., a function * $d \colon M \times M \rightarrow \mathbb{R}$
such that for any * $x, y, z \in M$, the following holds:[1]

* $d(x,y) \ge 0$     (non-negative),
* $d(x,y) = 0\, \iff x = y\,$     (identity of indiscernibles),
* $d(x,y) = d(y,x)\,$     (symmetry) and
* $d(x,z) \le d(x,y) + d(y,z) $    (triangle inequality) .

Examples of metric spaces include  [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance), [Manhattan distance](https://en.wikipedia.org/wiki/Taxicab_geometry), [Chebyshev distance](https://en.wikipedia.org/wiki/Chebyshev_distance), the [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) and many others.

### Vector dot product

*Algebraic definition*

The dot product of two vectors A = [A1, A2, ..., An] and B = [B1, B2, ..., Bn] is defined as:

$$ \mathbf{A}\cdot \mathbf{B} = \sum_{i=1}^n A_iB_i = A_1B_1 + A_2B_2 + \cdots + A_nB_n $$

*Geometric definition*

he magnitude of a vector A is denoted by  \left\| \mathbf{A} \right\| . The dot product of two Euclidean vectors A and B is:

$$\mathbf A \cdot \mathbf B = \left\| \mathbf A \right\| \, \left\| \mathbf B \right\| \cos \theta ,$$

where $\theta$ is the angle between A and B.

### Orthogonal and orthonormal

Two vectors are orthogonal if they are perpendicular, i.e., they form a right angle. Two vectors orthonormal if they are orthogonal and unit vectors. Orthogonal vectors in n-space if their dot product equals zero. That is, if A and B are orthogonal, then the angle between them is 90° and $\mathbf A \cdot \mathbf B = 0 .$

# Eigenvalues and eigenvectors

In linear algebra, an eigenvector or characteristic vector of a square matrix is a vector that does not change its direction under the associated linear transformation. In other words—if v is a vector that is not zero, then it is an eigenvector of a square matrix A if Av is a scalar multiple of v. This condition could be written as the equation

$$A\vec{v} = \lambda \vec{v}$$

where $\lambda$ is a number (also called a scalar) known as the eigenvalue or characteristic value associated with the eigenvector $\vec{v}$.

In this shear mapping the red arrow changes direction but the blue arrow does not. The blue arrow is an eigenvector of this shear mapping because it doesn't change direction, and since its length is unchanged, its eigenvalue is 1.

For example, Consider n-dimensional vectors that are formed as a list of n real numbers, such as the three dimensional vectors,

$$
\mathbf{u} = \begin{Bmatrix}1\\3\\4\end{Bmatrix}\quad\mbox{and}\quad \mathbf{v} = \begin{Bmatrix}-20\\-60\\-80\end{Bmatrix}. $$

These vectors are said to be scalar multiples of each other, also parallel or collinear, if there is a scalar λ, such that
\mathbf{u}=\lambda\mathbf{v}.

In this case $\lambda\$ = −1/20.

Now consider the linear transformation of n-dimensional vectors defined by an n×n matrix A, that is,

$$
 A\mathbf{v}=\mathbf{w},
or
\begin{bmatrix} A_{1,1} & A_{1,2} & \ldots & A_{1,n} \\
A_{2,1} & A_{2,2} & \ldots & A_{2,n} \\
\vdots &  \vdots &  \ddots &  \vdots \\
A_{n,1} & A_{n,2} & \ldots & A_{n,n} \\
\end{bmatrix}
\begin{Bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{Bmatrix} = \begin{Bmatrix} w_1 \\ w_2 \\ \vdots \\ w_n \end{Bmatrix}
where, for each index i,
 w_i = A_{i,1} v_1 + A_{i,2} v_2 + \cdots + A_{i,n} v_n = \sum_{j = 1}^{n} A_{i,j} v_j.

 $$
If it occurs that w and v are scalar multiples, that is if
$A\mathbf{v}=\lambda\mathbf{v}$,
then v is an eigenvector of the linear transformation A and the scale factor $\lambda\$ is the eigenvalue corresponding to that eigenvector.

Matrix A acts by stretching the vector x, not changing its direction, so x is an eigenvector of A.

from [Eigenvalues and eigenvectors - Wikipedia](https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors)

# PCA: Principal Component Analyses

[Principal component analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.

In Principal component analysis the feature selection is finding a set of linearly uncorrelated vectors (basis selection) of maximal variance. That is, the transformation is defined in such a way that the first principal component has the largest possible variance and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to the preceding components.

*First component*

The first loading vector $\vec{w}$ thus has to satisfy

$$
\mathbf{w}_{(1)}
 = \underset{\Vert \mathbf{w} \Vert = 1}{\operatorname{\arg\,max}}\,\left\{ \sum_i \left(t_1\right)^2_{(i)} \right\}
 = \underset{\Vert \mathbf{w} \Vert = 1}{\operatorname{\arg\,max}}\,\left\{ \sum_i \left(\mathbf{x}_{(i)} \cdot \mathbf{w} \right)^2 \right\} $$

Equivalently, writing this in matrix form gives

$$
\mathbf{w}_{(1)}
 = \underset{\Vert \mathbf{w} \Vert = 1}{\operatorname{\arg\,max}}\, \{ \Vert \mathbf{Xw} \Vert^2 \}
 = \underset{\Vert \mathbf{w} \Vert = 1}{\operatorname{\arg\,max}}\, \left\{ \mathbf{w}^T \mathbf{X}^T \mathbf{X w} \right\} $$

Since $\vec{w}$ has been defined to be a unit vector, it equivalently also satisfies

$$
\mathbf{w}_{(1)} = {\operatorname{\arg\,max}}\, \left\{ \frac{\mathbf{w}^T\mathbf{X}^T \mathbf{X w}}{\mathbf{w}^T \mathbf{w}} \right\}
$$

The first component is an eigenvector that maximizes the sum of squares

$$(\mathbf{Yv}_1)^\top \mathbf{Yv}_1$$

$\mathbf{v}_1$ is referred to as the _first principal component_ (PC). Also referred as  _first eigenvector_, $\mathbf{Yv}_1$ are the projections or coordinates or eigenvalues

*Further components*

The kth component can be found by subtracting the first k − 1 principal components from X:

$$
\mathbf{\hat{X}}_{k} = \mathbf{X} - \sum_{s = 1}^{k - 1} \mathbf{X} \mathbf{w}_{(s)} \mathbf{w}_{(s)}^{\rm T} $$

and then finding the loading vector which extracts the maximum variance from this new data matrix

$$
\mathbf{w}_{(k)} = \underset{\Vert \mathbf{w} \Vert = 1}{\operatorname{arg\,max}} \left\{ \Vert \mathbf{\hat{X}}_{k} \mathbf{w} \Vert^2 \right\} = {\operatorname{\arg\,max}}\, \left\{ \tfrac{\mathbf{w}^T\mathbf{\hat{X}}_{k}^T \mathbf{\hat{X}}_{k} \mathbf{w}}{\mathbf{w}^T \mathbf{w}} \right\} $$

It turns out that this gives the remaining eigenvectors of $X^TX$, with the maximum values for the quantity in brackets given by their corresponding eigenvalues.

The kth  is the vector that

$$ \mathbf{v}_{k}^\top \mathbf{v}_{k}=1$$

$$ \mathbf{v}_{k}^\top \mathbf{v}_{k-1}=0$$

and maximizes  $$(\mathbf{rv}_{k})^\top \mathbf{rv}_{k}$$

### Properties of Principal Components

* The principal components are eigenvectors
* Maximizes variance in order of the components
* They are orthogonal (and form a basis)
* If use all of the principal components  we can get lossless reconstruction
* N diminsions to M diminsions (each diminsion has an eigenvalue
the order (highest eigenvalues have highest  variance)
i.e. can "throw away" the lowest eigenvalues.
if eigenvalue has 0 value can throw it away without cost.
* We can readjust to origin 0
* There are very fast algorithms to compute PCA

*Nota bene*
Note that there may be oroblems of interpretation of the components.
Note that a low variance dimension may be important

## LDA: Linear Discriminant Analysis

LDA: Linear Discriminant Analysis

An alternative method to calculate eigenvectors from covariance matrix
[Linear discriminant analysis (LDA)](https://en.wikipedia.org/wiki/Linear_discriminant_analysis) is a generalization of Fisher's linear discriminant, a method used in statistics, pattern recognition and machine learning to find a linear combination of features that characterizes or separates two or more classes of objects or events.

Singular Value Decomposition

In linear algebra, the [singular value decomposition (SVD)](https://en.wikipedia.org/wiki/Singular_value_decomposition) is a factorization of a real or complex matrix. A  a matrix decomposition or matrix factorization is a factorization of a matrix into a product of matrices.

If $\mathbf{M}$ is a m × n matrix whose entries come from $\mathbb{R}$, then the SVD  a factorization, called a singular value decomposition of  $\mathbf{M}$ , of the form $\mathbf{M=UDV}^\top$ gives the factors in columns of $\mathbf{V}$.

Note that,
$\mathbf{D}$ is a m × n diagonal matrix with non-negative real numbers on the diagonal, and
$\mathbf{U}$  is an m × m, and $\mathbf{V}$ is an n × n, unitary matrix over $\mathbb{R}$.

The diagonal entries, $\sigma_{i}$, of $D$ are known as the singular values of $\mathbf{M}$. The singular values are usually listed in descending order.

Note that [Eigendecomposition](https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix) is a form of matrix factorization.

$A=VDV^{-1}$, where D is a diagonal matrix formed from the eigenvalues of A, and the columns of V are the corresponding eigenvectors of A.

### Differences between Principal Component Analysis and Singular-value decomposition

The singular value decomposition and the eigendecomposition are closely related.  PCA viewpoint requires that one compute the eigenvalues and eigenvectors of the covariance matrix, which is the product $\mathbf{X}\mathbf{X^T}$, where $\mathbf{X}$ is the data matrix and SVD constructs the covariance matrix from this decomposition

$$\mathbf{X}\mathbf{X^T}= \mathbf{(UDV^T)}\mathbf{(UDV^T)}^T$$

$$\mathbf{X}\mathbf{X^T}= \mathbf{(UDV^T)}\mathbf{(VDU^T)}$$

and since $\mathbf{V}$ is an orthogonal matrix $\mathbf{V^TV}=\mathbf{I}$),

$$\mathbf{X}\mathbf{X^T}= \mathbf{U}\mathbf{D}^2\mathbf{U^T}$$

That is,
- The left-singular vectors of $\mathbf{M}$ are eigenvectors of $\mathbf{M}\mathbf{M^*}$ .
- The right-singular vectors of  $\mathbf{M}$ are eigenvectors of $\mathbf{M^*}\mathbf{M}$.
- The non-zero singular values of $\mathbf{M}$  (found on the diagonal entries of $\mathbf{D}$) are the square roots of the non-zero eigenvalues of both $\mathbf{M^*}\mathbf{M}$ and $\mathbf{M}\mathbf{M^*}$.

## Regularization

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, is a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting.

- Regularization is a method for "constraining" or "regularizing" the **size of the coefficients**, thus "shrinking" them towards zero.
- It reduces model variance and thus **minimizes overfitting**.
- If the model is too complex, it tends to reduce variance more than it increases bias, resulting in a model that is **more likely to generalize**.

### Regularization in linear regression

For a normal linear regression model, we estimate the coefficients using the least squares criterion, which **minimizes the residual sum of squares (RSS):**

For a regularized linear regression model, we **minimize the sum of RSS and a "penalty term"** that penalizes coefficient size.

**Ridge regression** (or "L2 regularization") minimizes: $$\text{RSS} + \alpha \sum_{j=1}^p \beta_j^2$$

**Lasso regression** (or "L1 regularization") minimizes: $$\text{RSS} + \alpha \sum_{j=1}^p |\beta_j|$$

- $p$ is the **number of features**
- $\beta_j$ is a **model coefficient**
- $\alpha$ is a **tuning parameter:**
    - A tiny $\alpha$ imposes no penalty on the coefficient size, and is equivalent to a normal linear regression model.
    - Increasing the $\alpha$ penalizes the coefficients and thus shrinks them.

## Cross Validation

[Cross validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)) is a model validation technique for assessing how the results of a statistical analysis will generalize to an independent data set. It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice.

**k-fold cross-validation**

In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data. The k results from the folds can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used,[7] but in general k remains an unfixed parameter.

For example, setting k = 2 results in 2-fold cross-validation. In 2-fold cross-validation, we randomly shuffle the dataset into two sets d0 and d1, so that both sets are equal size (this is usually implemented by shuffling the data array and then splitting it in two). We then train on d0 and test on d1, followed by training on d1 and testing on d0.

When k = n (the number of observations), the k-fold cross-validation is exactly the leave-one-out cross-validation.

In stratified k-fold cross-validation, the folds are selected so that the mean response value is approximately equal in all the folds. In the case of a dichotomous classification, this means that each fold contains roughly the same proportions of the two types of class labels.

## Time Series

A [time series](https://en.wikipedia.org/wiki/Time_series) is a sequence of data points, typically consisting of successive measurements made over a time interval. This is a very common type of data as we frequently measure how something varies over time.  usually a time series is an ordered sequence of values of a variable at equally spaced time intervals; but methods exist to deal with irregular sampling.

Examples of time series include:

* Stock Market
* The change of weather
* An electrocardiogram (EKG or ECG), that is, the electrical activity of your heart.
* The popularity of a celebirty or politician
* Economic Forecasting

- from https://commons.wikimedia.org/wiki/File:Gold_Spot_Price_per_Gram_from_Jan_1971_to_Jan_2012.svg

Univariate (bivariate, multivariate) time series: collection of observations of one (two, several) state variables, that are made in sequential momentsin time.

The [sampling](https://en.wikipedia.org/wiki/Sampling_(signal_processing)) frequency (or sample rate) is the number of samples per unit of time. (e.g once per minute).

- from https://commons.wikimedia.org/wiki/File:Signal_Sampling.png

### Nyquist criterion

Nyquist criterion, or [Nyquist frequency](https://en.wikipedia.org/wiki/Nyquist_frequency), named after electronic engineer [Harry Nyquist](https://en.wikipedia.org/wiki/Harry_Nyquist), is half of the sampling rate of a discrete signal processing system. In order to correctly determine the frequency spectrum of a signal, the signal must be measured at least twice per period.

### Reconstruction

Reconstruction is the opposite of sampling, the process by which a sampled signal is converted into a continuous signal.

Under-Sampling

### Components of Time Series

The pattern or behavior of the data in a time series can be made up of several components. Theoretically, any time series can be decomposed into:

*Trend*

The trend component accounts for the gradual shifting of the time series to relatively higher or lower values over a long period of time.

*Cyclical*

A regular pattern of sequences of values that go above and below the trend line is a cyclical component.

*Seasonal*

The seasonal component accounts for regular patterns of variability within certain time periods, such as a year. While seasons could be modeled using cyclical components. Often within-year, within-month, within-week or within-day cycles are treated as “seasonal” behavior.

*Irregular*

The irregular components represent erratic, unsystematic, ‘residual’ fluctuations. That is, noise.

### Stationary versus Non-stationary time series

Non-stationary time series or have means, variances and covariances that change over time. Whereas while stationary time series have up and down flucutions they have means, variances and covariances.  That is, a [stationary process](https://en.wikipedia.org/wiki/Stationary_process) is a stochastic process whose joint probability distribution does not change when shifted in time.

For example, we expect financial time series data, to be non-stationary since we would expect a deterministic upward trend in price.

A grown adults weight would more likely be stationary time series unless he or she fundenemtally changes their "state".

*Two simulated time series processes, one stationary the other non-stationary.*
- from https://commons.wikimedia.org/wiki/File:Stationarycomparison.png

### Smoothing Methods of Time Series

Smoothing a time series: to eliminate some of short-term fluctuations. Sometimes smoothing is done to remove cyclical or  seasonal fluctuations. (i.e. "deseasonalize"" a time series). Usually smoothing refers to methods for reducing of canceling the effect due to random variation. The simplest smoothing method is just the "simple" average of all past data.  Of course, the "simple" average or mean of all past observations is only a useful estimate for forecasting when there are no trends.

### Time Series Decomposition

Time Series Decomposition is a procedure to identify the component factors of a time series. That is, to create a model that expresses the time series variable Y in terms of the components T (trend), C (cycle), S (seasonal) and I (iregular).

## Time Series Analysis

Time series analysis generates a model that accounts an internal structure (such as autocorrelation, trend or seasonal variation) of a set of data points taken over time. This is very frequently used for forecasting. That is, predicting and future event in time based on a sequential historical sample.

### Autocorrelation

[Autocorrelation](https://en.wikipedia.org/wiki/Autocorrelation), also known as serial correlation or cross-autocorrelation, is the cross-correlation of a signal with itself at different points in time. It is the correlation between values of the process at different times. Informally, it is the similarity between observations as a function of the time lag between them. It is a mathematical tool for finding repeating patterns, such as the presence of a periodic signal obscured by noise.

Let X be some repeatable process, and i be some point in time after the start of that process. (i may be an integer for a discrete-time process or a real number for a continuous-time process.) Then $X_i$ is the value (or realization) produced by a given run of the process at time i. Suppose that the process is further known to have defined values for mean μi and variance σi2 for all times i. Then the definition of the autocorrelation between times s and t is

$$
R(s,t) = \frac{\operatorname{E}[(X_t - \mu_t)(X_s - \mu_s)]}{\sigma_t\sigma_s}\, ,
$$

where "E" is the expected value operator. Note that this expression is not well-defined for all time series or processes, because the variance may be zero (for a constant process) or infinite (for processes with distribution lacking well-behaved moments, such as certain types of power law).

### Correlogram

In the analysis of data, a [correlogram](https://en.wikipedia.org/wiki/Correlogram) is an image of correlation statistics. For example, in time series analysis, a correlogram, also known as an autocorrelation plot, is a plot of the sample autocorrelations $r_{h}, r_{h}$, versus $h$ (the time lags).

If cross-correlation is used, the result is called a cross-correlogram. The correlogram is a commonly used tool for checking randomness in a data set. This randomness is ascertained by computing autocorrelations for data values at varying time lags. If random, such autocorrelations should be near zero for any and all time-lag separations. If non-random, then one or more of the autocorrelations will be significantly non-zero.

[By M. W. Toews - Own work, CC BY 4.0](https://commons.wikimedia.org/w/index.php?curid=2379384)

### Naive & Simple Averaging

Naive or "simple" averaging is just the mean (or median) of all past data.  Of course, the "simple" average or mean of all past observations is only a useful estimate for forecasting when there are no trends.

For example, if we have prices are $p_M, p_{M-1},\dots,p_{M-(n-1)}$ then the formula is

$$
\textit{SMA} = { p_M + p_{M-1} + \cdots + p_{M-(n-1)} \over n }
$$

### Moving Averages

A moving average (rolling average or running average) is an average that is updated for a window or history of n events. This is sometimes called the arithmetic moving average the most recent n data values. For an an equally weighted average of the sequence of n values $x_1. \ldots, x_n$ up to the current time:
$$
\textit{MA}_n = {{x_1 + \cdots + x_n} \over n}\,.
$$

### Weighted Moving Averages

A weighted moving average places more weight on recent observations.  Sum of the weights needs to equal 1. For example. in technical analysis of financial data, a weighted moving average (WMA) often uses weights that decrease in arithmetical progression. In an n-day WMA the latest day has weight n, the second latest n − 1, etc., down to one.

$$
\text{WMA}_{M} = { n p_{M} + (n-1) p_{M-1} + \cdots + 2 p_{(M-n+2)} + p_{(M-n+1)} \over n + (n-1) + \cdots + 2 + 1}
$$

That is, the denominator is a triangle number equal to $\frac{n(n+1)}{2}$.

There are many other methods for weighting the importance of more recent observations.

"Weighted moving average weights N=15". Licensed under CC BY-SA 3.0 via Commons

### Exponential moving average

Exponential moving average (also called Single Exponential Smoothing) continually revises a forecast in light of more recent experiences.  Averaging (smoothing) past values of a series in a decreasing (exponential) manner.

The EMA for a series Y may be calculated recursively:

$$
S_1   Y_1 \quad for  \quad t > 1, S_{t} = \alpha \cdot Y_{t} + (1-\alpha) \cdot S_{t-1}
$$

Where:

* The coefficient $\alpha$ represents the degree of weighting decrease, a constant smoothing factor between 0 and 1. A higher  $\alpha$  discounts older observations faster.
* $Y_t$ is the value at a time period t.
* $S_t$ is the value of the EMA at any time period t.

Since the base values of the recursive intial call is undefined. The priming of values such as $S_1$ may be initialized in a number of different ways, most commonly by setting $S_1$ to $Y_1$, or by setting  $S_1$to an average of the first few observations.

"Exponential moving average weights N=15". Licensed under CC BY-SA 3.0 via Commons

### Double exponential smoothing

Holt-Winters method for exponential smoothing extended Holt’s method to capture seasonality. This is sometimes called "Double exponential smoothing." The Holt-Winters seasonal method comprises the forecast equation and two smoothing equations. e use {$s_t$} to represent the smoothed value for time t, and {$b_t$} is our best estimate of the trend at time t. The output of the algorithm is now written as Ft+m, an estimate of the value of x at time t+m, m>0 based on the raw data up to time t. Double exponential smoothing is given by the formulas

$$
\begin{align}
s_1& = x_1\\
b_1& = x_1 - x_0\\
\end{align}
$$

And for t > 1 by

$$
\begin{align}
s_{t}& = \alpha x_{t} + (1-\alpha)(s_{t-1} + b_{t-1})\\
b_{t}& = \beta (s_t - s_{t-1}) + (1-\beta)b_{t-1}\\
\end{align}
$$

where α is the data smoothing factor, 0 < $\alpha$ < 1, and β is the trend smoothing factor, 0 < $\beta$ < 1.

### Triple exponential smoothing

Triple exponential smoothing account seasonal changes as well as trends using three smoothing equations — one for the level, one for trend, and one for the seasonal component denoted, with smoothing parameters $\alpha, \beta,$ and $\gamma$. We use a variable $m$ to denote the period of the seasonality, i.e., the number of seasons in a year. For example, for quarterly data m=4, and for monthly data m=12.

There are two variations to this method that differ in the nature of the seasonal component. The additive method is preferred when the seasonal variations are roughly constant through the series, while the multiplicative method is preferred when the seasonal variations are changing proportional to the level of the series.

## Autoregressive integrated moving average (ARIMA)

An autoregressive integrated moving average (ARIMA or ARMA) model combines an autoregressive component with a moving average component in to a single model.

An [autoregressive integrated moving average (ARIMA or ARMA)](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average) model is a generalization of an autoregressive moving average (ARMA) model. These models are fitted to time series data either to better understand the data or to predict future points in the series (forecasting). They are applied in some cases where data show evidence of non-stationarity, where an initial differencing step (corresponding to the "integrated" part of the model) can be applied to reduce the non-stationarity.

Non-seasonal ARIMA models are generally denoted $ARIMA(p, d, q)$ where parameters $p, d, and q$ are non-negative integers, $p$ is the order of the Autoregressive model, $d$ is the degree of differencing, and $q$ is the order of the Moving-average model.  The he number of differences $d$ is determined using repeated statistical tests. The values of $p$ and $q$ are then chosen by minimizing the AICc after differencing the data $d$ times.

The ARIMA model uses an iterative three-stage modeling approach:

Model identification and model selection: making sure that the variables are stationary, identifying seasonality in the dependent series (seasonally differencing it if necessary), and using plots of the autocorrelation and partial autocorrelation functions of the dependent time series to decide which (if any) autoregressive or moving average component should be used in the model.

Parameter estimation using computation algorithms to arrive at coefficients that best fit the selected ARIMA model. The most common methods use maximum likelihood estimation or non-linear least-squares estimation.

Model checking by testing whether the estimated model conforms to the specifications of a stationary univariate process. In particular, the residuals should be independent of each other and constant in mean and variance over time. (Plotting the mean and variance of residuals over time and performing a Ljung-Box test or plotting autocorrelation and partial autocorrelation of the residuals are helpful to identify misspecification.) If the estimation is inadequate, we have to return to step one and attempt to build a better model.

## Linear Time-Series Model

Using regression we can model and forecast the trend in time series data by including the time as a predictor variable. Time series processes are often described by multiple linear regression (MLR) models of the form:

$$y_t = X_t \beta + e_t,$$

Note that the above equation differs from our usual regression in that the predictor variables are a function of time, $t$. As oppossed to the more familair form below:

$$ Y = \beta X + e $$

where  $y_t$ is an observed response and  $X_t$ includes columns for contemporaneous values of observable predictors. The partial regression coefficients in  $\beta$ represent the marginal contributions of individual predictors to the variation in  $y_t$ when all of the other predictors are held fixed. A common feature of time series data is a trend. Regressing non-stationary time series can lead to spurious regressions.

The error term $\varepsilon_i$ still has the following assumptions:

* have mean zero; otherwise the forecasts will be systematically biased.
* statistical independence of the errors (in particular, no correlation between consecutive errors in the case of time series data).
* homoscedasticity (constant variance) of the errors.
* normality of the error distribution.

## Text Mining

[Text mining](https://en.wikipedia.org/wiki/Text_mining), also called text analytics, is the process of deriving high-quality information from text. Given the vast amount of text on the Internet, text mining is one of the most important research areas in machine learning. Text mining includes:

* [Information retrieval](https://en.wikipedia.org/wiki/Information_retrieval) - the process of obtaining structured data from free text.
* [Entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition) - identification of nouns (e.g. known people, places and things) in text.
* Fact extraction - identification of associations among entities and other information in text.
* [Sentiment analysis](https://en.wikipedia.org/wiki/Sentiment_analysis) is the assocaition of opinion and a topic within text.
* [Topic modeling](https://en.wikipedia.org/wiki/Topic_model) - identification of "topics" in text.
* Automated Tagging - a process to associate free-form keywords (not necessarily in the text) to index and search text.

##  Text corpora and dictionaries

In linguistics, a corpus (plural corpora) or text corpus is a large and structured set of texts. They help annotate words.

* [corpus.byu.edu](http://corpus.byu.edu/) includes:
     + Hansard Corpus (British Parliament), 1.6 billion words
     + Global Web-Based English (GloWbE), 1.9 billion words
     + Corpus of Contemporary American English (COCA), 450 million words
     + TIME Magazine Corpus, 100 million words
     + Corpus of American Soap Operas, 100 million words
     + British National Corpus (BYU-BNC), 100 million words
     + Strathy Corpus (Canada), 100 million words
* The [British National Corpus](http://www.natcorp.ox.ac.uk/), extracts from 4124 modern British English texts of all kinds, both spoken and written; over 100 million words.
* The [Brown University Corpus](http://www.hit.uib.no/icame/brown/bcm.html): Approximately 1,000,000 words of American written English dating from 1960. The genre categories are parallel to those of the LOB corpus.
* The [LOB Corpus](http://clu.uni.no/icame/manuals/) (The Lancaster-Oslo/Bergen Corpus) is ) is a million-word collection of British English texts which was compiled in the 1970s.
* [The Kolhapur Corpus](http://www.essex.ac.uk/linguistics/external/clmt/w3c/corpus_ling/content/corpora/list/private/kolhapur.html): Approximately 1,000,000 words of Indian written English dating from 1978.
* The (Cambridge International Corpus)[http://uk.cambridge.org/elt/corpus] is a multi-billion word corpus of English language (containing both text corpus and spoken corpus)
* [The Longman-Lancaster Corpus](http://www.pearsonlongman.com/dictionaries/corpus/lancaster.html): Approximately 14.5 million words of written English from various geographical locations in the English-speaking world and of various dates and text types.* [WordNet](http://wordnet.princeton.edu/) is a lexical database of English nuns, verbs, adjectives and adverbs which are grouped into sets of cognitive synonyms (synsets). The WordNet synsets are further characterized by hyperonymy, hyponymy or ISA relationships. We downloaded the WordNet database files and parsed them. Permission to use, copy, modify and distribute WordNet for any purpose and without fee or royalty is hereby granted, WordNet provided by WordNet as long as proper attribution is given to WordNet and any derivative products don’t use the WordNet trademark.
* [PubMed/Medline](http://www.ncbi.nlm.nih.gov/pubmed) comprises more than 25 million citations for biomedical literature. PubMed XML Data Retrieved from [http://www.nlm.nih.gov/databases/journal.html](http://www.nlm.nih.gov/databases/journal.html). You need to regiester with the National Library of Medicine to download the XML files.
* [arXiv](http://arxiv.org/) is an  archive with over 100000 articles in physics, 10000 in mathematics, and 1000 in computer science. arXiv Bulk Data Access Retrieved from [http://arxiv.org/help/bulk_data](http://arxiv.org/help/bulk_data)
* AG's news corpus is AG's corpus of news articles. Retrieved from [http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)
* Last.fm music tags can be retrieved from [http://www.last.fm/charts/toptags](http://www.last.fm/charts/toptags )
* Spambase can be retrieved from [http://archive.ics.uci.edu/ml/datasets/Spambase](http://archive.ics.uci.edu/ml/datasets/Spambase)
* Wikipedia. The  Wikipedia Data Dump can be retrieved from [http://en.wikipedia.org/wiki/Wikipedia:Database_download](http://en.wikipedia.org/wiki/Wikipedia:Database_download)

###  Unstructured text data

*  [Common Crawl](http://commoncrawl.org/) is a openly accessible web crawl data that is freely available. [164] As of April 2013 the crawl has 6 billion pages and associated metadata. The crawl data is stored on Amazon’s Public Data Sets, allowing it to be directly accessed for map-reduce processing in EC2. Common Crawl Retrieved from [http://commoncrawl.org/](http://commoncrawl.org/)

* Twitter data.
    + Twitter Search API Retrieved from [https://dev.twitter.com/docs/api/1/get/search](https://dev.twitter.com/docs/api/1/get/search)
    + Twitter Streaming APIs.  Retrieved from [https://dev.twitter.com/docs/streaming-apis](https://dev.twitter.com/docs/streaming-apis)
    + Twitter“Fire hose” real-time stream. See [https://dev.twitter.com/streaming/firehose](https://dev.twitter.com/streaming/firehose)

*  Instagram
    + API - Instagram [https://instagram.com/developer/](https://instagram.com/developer/)

###  Writing a web crawler

A Web crawler (also known as a Web spider or Web robot or  bot) is a pscript which browses the World Wide Web and extracts web pages and links.
```
urls=<list of urls>
while (urls)
{
  * request url
  * request document
  * store text for later processing
  * parse document for links
}
```

*Useful web crawling libraries in R*

*  library(regex) - Regular Expressions as used in R
*  library(httr) - Tools for Working with URLs and HTTP
*  library(XML) - XML Parser and Generator
*  library(RCurl) - RCurl: General Network (HTTP/FTP/...) Client Interface for R
*  library(jsonlite) - JSON Parser and Generator for R
*  library(stringr) - Simple, Consistent Wrappers for Common String Operations

## N-grams

An [n-gram](https://en.wikipedia.org/wiki/N-gram) is a contiguous sequence of n items from a given sequence of text or speech. An n-gram of size 1 is referred to as a "unigram"; size 2 is a "bigram" (or, less commonly, a "digram"); size 3 is a "trigram". Larger sizes are sometimes referred to by the value of n, e.g., "four-gram", "five-gram", and so on.

for example, bugaboo, nerd, student, data and new are one-grams. Data science, New York are two-grams. Note that New York means something different the the one grams new, and York seperately.

Note that if we take a sentence, say

"You have brains in your head. You have feet in your shoes. You can steer yourself in any direction you choose. You're on your own, and you know what you know. And you are the guy who'll decide where to go."

- Dr. Seuss

Most of the two-grams aren't meaningful, that is, "You have," "have brains," "brains in" don't make sense out of context.

## Tagging and Hashtags

Tagging is a process in which end users use free-form keywords to manually index content in an organic and distributed manner. The popularity of tagging has led some to claim that it is the primary classification scheme of the Internet. A tag can be thought of as an informative keyword. A user is very unlikely to tag an article with a word like “this” because it conveys very little information. Rather, they’ll often tag with a subject or sentiment.

Problems with tagging are well-known.  Users often present idiosyncrasies, inaccuracies, inconsistencies, and other irregularities when tagging. Specifically, four areas are critical to tagging. The first three areas are straightforward enough:

1. tag misspelling;
2. tag heterogeneity, (that is, different tags denoting the same content, such as “Ziagen” and “abacavir sulfate,” which both refer to the same drug);
3. tag polysemy (i.e. identical tags that denote different meanings, such as, Apple may refer to fruit or a company. and;
4. semantic annotation of tags (i.e. abacavir sulfate is a drug).

An important area of research in text mining is call  “semantic enrichment” is a particularly difficult problem. Lexical resources are often used to annotate terms. For example, lexical databases such as WordNet is often used as a source of tag annotation.

*##hashtags*

The social tagging in is done by placing a hash mark in front of a word or phrase, such as ##BCSM, ##Lyphoma, ##BrainTumorThursday, ##BreastCancer, ##Infertility, ##Diabetes, ##lymphoedema, ##RareDiseaseDay, ##RareDisease, ##ADHD, ##Anorexia, ##MultipleSclerosis. On social media sites (such as Twitter) a word or phrase preceded by a hash or pound sign (##) and used to identify messages on a specific topic. Hastags are essentially tags that in within text and annotated.  As such they are easy to extract from text.

*Stop words*

In text mining, [stop words](https://en.wikipedia.org/wiki/Stop_words) are words which are filtered out becuase they'll interfer with text analysis. Stop words usually refer to the most common words in a language, such as *the*, *is*, *at*, *which*, and on.

## Regular expressions

Regular expressions (or RE or regex or regexp or rational expression) is a sequence of characters that define a search pattern, mainly for use in pattern matching with strings, or string matching.

Cerain characters have special meaning in regular expressions.

[] - A pair of brackets is used to indicate a set of characters.
'\' - Either escapes special character or signals a special sequence.
?	- The question mark indicates there is zero or one of the preceding
'*'	- The asterisk indicates there is zero or more of the preceding element  .
+	The plus sign indicates there is one or more of the preceding element.
'^' (Caret.) Matches the start of the string.
'$' Matches the end of the string.
{m} (Braces) Specifies that exactly m copies of the previous RE should be matched.

For example, the regexp $[A-Za-z]+$ matches a seuqnece of at least one upper or lower case letters.  The regexp $^[ ]+A-Za-z0-9._-]+@[[A-Za-z0-9.-]+$[ ]+$ matches an e-mail pattern with starting and ending white spaces. There are many excellent books that describe regular expressions in detail.

## Term-Document matrices

A [term-document matrix](https://en.wikipedia.org/wiki/Document-term_matrix) is a matrix where the rows correspond to documents in the collection and columns correspond to terms. Creating term-document matrices are created using the [R text mining package tm](https://cran.r-project.org/web/packages/tm/index.html).

D1 = "I love R"
D2 = "I love ice-cream"

then the document-term matrix would be:

$$
TermDocument=
  \begin{bmatrix}
      & I & love & R & ice-cream \\
    D1 & 1 & 1 & 1 & 0 \\
    D2 & 1 & 1 & 0 & 1
  \end{bmatrix}
$$

which shows which documents contain which terms and how many times they appear.

## Frequency signatures

For processing larger amounts of text, tag/word counts can be ineffecient using term-document matrices as these are typically very sparse matrices. Especially when one has a large dictionary of tags/words.

For larger data we used a “frequency signature” approach to convert a [bag-of-words](https://en.wikipedia.org/wiki/Bag-of-words_model) output to a format that we can use to calculate tag co-occurrence associations and mutual information. Frequency signatures are described in detail in Stefan Evert’s PhD dissertation “The Statistics of Word Cooccurrences Word Pairs and Collocations.”

To calculate tag co-occurrence associations and mutual information for two tags, A and B, we need four items of data. The co-occurrence count of A and B, the count of A but not B, the count of B but not A, and the total number of tags in a corpus. This co-occurrence frequency data for a word pair (A,B) are usually organized in a contingency table show below. The contingency table stores the observed frequencies $O_{11} … O_{22}$. The table below (adapted from Evert’s dissertation) shows an observed contingency table.

Contingency Tables

*Contingency table : $O_{11}$ is co-occurrence count of A and B, $O_{12}$ is the count of A but not B, $O_{21}$  is the count of B but not A, and $O_{22}$ is the count of not B and not A.*

However, while the co-occurrence count of A and B, and the total number of tags in a corpus are efficiently and easily counted the count of A but not B, the count of B but not A are tricky and computationally expensive. The insight and advantage of frequency signatures is that they calculate the count of A but not B, the count of B but not A by just counting A and B and the co-occurrence count of A and B. That is, the count of A but not B is equal to count of A minus the co-occurrence count of A and B. Likewise, the count of B but not A is equal to count of B minus the co-occurrence count of A and B.

The frequency signature of a tag pair (A, B) is usually written as $(f, f_1, f_2,N)$. Where $f$ is the co-occurrence count of A and B, $f_1$ is the count of A but not B, $f_2$ is the count of B but not A, and N is the total counts. Notice that the observed frequencies $O_{11}, ..., O_{22}$ can be directly calculated from the frequency signature by the equations below:

* $O_{11} = f$
* $O_{12} = f_1 − f$
* $O_{21} = f_2 – f$
* $O_{22} = N − f_1 − f_2 + f$

Generating all of the data tag co-occurrence association and mutual information calculations using this approach can be generated using a single pass of the data and two associative arrays; one of the tag counts and another for the tag co-occurrence counts.

Calculating Associations and Mutual Information from Frequency Signatures

Evert shows the many association and mutual information statistics can be calculated from the observed frequencies $O_{11}, ..., O_{22}$ if we can generate the expected frequencies $E_{11}, ..., E_{22}$. The table below (adapted from Evert’s dissertation) shows the expected versus observed contingency tables.

*Frequency Signatures*

The sum of all four observed frequencies (called the sample size N) is equal to the total number of pair tokens extracted from the corpus. R1 and R2 are the row totals of the observed contingency table, while C1 and C2 are the corresponding column totals. The expected frequencies can be directly calculated from observed frequencies $O_{11}, ..., O_{22}$ by the equations below:

* $R1 = O_{11} + O_{12}$
* $R2 = O_{21} + O_{22}$
* $C1 = O_{11} + O_{21}$
* $C2 = O_{12} + O_{22}$
* $N = O_{11} + O_{12} + O_{12} + O_{22}$

Evert went on to show that several association measures can be easily calculated once one has the expected and observed contingency tables. For example, the pointwise mutual information (MI) is calculated by below.

$pointwise \quad mutual \quad information \quad MI=\ln\frac{O_{11}}{E_{11}}$

The Likelihood measures that can be calculated using the expected and observed contingency tables are:

* Multinomial-likelihood
* Binomial-likelihood
* Poisson-likelihood
* Poisson-Stirling approximation
* Hypergeometric-likelihood

The exact hypothesis tests that can be calculated using the expected and observed contingency tables are:

* binomial test
* Poisson test
* Fisher's exact test

The asymptotic hypothesis tests that can be calculated using the expected and observed contingency tables are:

* z-score
* Yates' continuity correction
* t-score (which compares O11 and E11 as random variates)    * Pearson's chi-squared test
* Dunning's log-likelihood (a likelihood ratio test)

The measures from information theory that can be calculated using the expected and observed contingency tables are:

* MI (mutual information, mu-value)
* logarithmic odds-ratio logarithmic relative-risk
* Liddell's difference of proportions
* MS (minimum sensitivity)
* gmean (geometric mean) coefficient
* Dice coefficient (aka. "mutual expectation")
* Jaccard coefficient
* MIconf (a confidence-interval estimate for the mu-value)
* MI (pointwise mutual information)
* local-MI (contribution to average MI of all co-occurrences)
* average-MI (average MI between indicator variables)

Stefan Evert also developed a R library called [UCS toolkit](http://www.collocations.de/software.html) for the statistical analysis of co-occurrence data with association measures and their evaluation in a collocation extraction task.

## tf–idf

[Tf–idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) or term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document.

The tf-idf value increases proportionally to the number of times a word/tag appears in the document, but is offset by the frequency of the word/tag. It is a measure of jargon. If a word appears frequently in a document, it's important. Give the word a high score. But if a word appears in every documents, it's not specifc to a topic. Give the word a low score.

If we want to find jargon, that is, topic or subject specifc words/tags then this is a reasonable metric. Words like 'the', 'a', get low scores as the are in every document. While words/tags like 'Machine Learning', 'Twitter,' or 'Text Mining' get high scores since they are used a lot in specific contexts.

If with call $f(t,d)$ the raw frequency of a term in a document, i.e. the number of times that term t occurs in document d and $max(f(t,d))$ the maximum raw frequency of any term in the document, then the term frequency $tf(t,d)$ is:

$$
\mathrm{tf}(t,d) = 0.5 + \frac{0.5 \times \mathrm{f}(t, d)}{\max\{\mathrm{f}(t, d):t \in d\}}
$$

The inverse document frequency is a measure of how much information the word provides, It is the logarithmically scaled fraction of the documents that contain the word, obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient.

$$
 \mathrm{idf}(t, D) =  \log \frac{N}{|\{d \in D: t \in d\}|}
$$

with

* $N$: total number of documents in the corpus $N = {|D|}$
* $|\{d \in D: t \in d\}|$ : number of documents where the term  t  appears (i.e.,  $\mathrm{tf}(t,d) \neq 0$). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to adjust the denominator to add a single "pseudocount."

Mathematically the base of the log function does not matter for these purposes.

Then tf–idf is calculated as

$$\mathrm{tf-idf}(t,d,D) = \mathrm{tf}(t,d) \times \mathrm{idf}(t, D)$$

A high weight in tf–idf means a tag/word has high term frequency (in the given document) and a low document frequency of the term in the whole collection of documents.

For how this is done in R see [The tf-idf-Statistic For Keyword Extraction](http://www.r-bloggers.com/the-tf-idf-statistic-for-keyword-extraction/)

## Word entropy and entropy rate

In information theory, entropy is also a measure of the uncertainty in a random variable. Like tf–idf, entropy quantifies the expected value of the information contained in a message (or the expected value of the information of the probability distribution). Typically this is expressed in the number of ‘bits’ or ‘nats’ that are required to encode a given message.

In this sense entropy can be used to estimate (like tf–idf) how much information is in a word or tag. Entropy can also be used to estimate the generating probability distribution for a text document or corpus. The entropy of many languages has been determined. English has 1.65 bits per word, French has 3.02 bits per word, German has 1.08 bits per word, and Spanish has 1.97 bits per word. Given the probability density function of word entropies and the average bits per word of a single tweet we could then assign probabilities that it is English, French, German, or Spanish.

### Entropy rate

The entropy rate (or mean entropy rate) describes the limiting entropy over an entire probability distribution.  This can be thought of as the average entropy over a sufficiently long realization of a stochastic process, whereas the entropy is relevant to a single random variable at a given point in time.

In statistics, ergodicity describes a random process wherein the average time for one sufficiently long realization of events is the same as the ensemble average. That is, the ensemble’s statistical properties (such as its mean or entropy) can be deduced from a single, sufficiently long sample of the process. In other words, there are long-term invariant measures that describe the asymptotic properties of the underlying probability distribution, and they can be measured by following any single reprehensive portion if followed long enough. For example, if I look at two particles in an ergodic system at any time, those particles may have very different states; but if I follow those particles long enough, they become statistically indistinguishable from one another. This means that statistical properties of the entire system can be deduced from a single sample of the process if followed for a sufficiently long time.

Stationarity is the property of a random process which guarantees that the aggregate statistical properties of the probability density function, such as the mean value, its moments and variance, remain the same at every point in time. A stationary process, therefore, is one whose probability distribution is the same at all times. Its statistical properties cannot necessarily be deduced from a single sample of the process. There are stochastic processes that exhibit both stationarity and ergodicity called stationary ergodic processes. These are random processes that will not change their statistical properties with time; hence, the properties, including the disorder (entropy) of the system, can be deduced from a single, sufficiently long sample realization of the process. There are weaker forms of the stationary condition in which the first- and second-order moments (that is, the mean and variance) of a stochastic process are constant but other properties of the probability density function can vary. Likewise, there are stationary stochastic processes that are not themselves ergodic but are composed of a mixture of ergodic components.

##### Rényi entropies

The [Rényi entropies](https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy) generalize the Shannon entropy, the Hartley entropy, the min-entropy, and the collision entropy. As such, these entropies as an ensemble are often called the Rényi entropies (or the Rényi entropy, even though this usually refers to a class of entropies). The difference between these entropies is in the respective value for each of an order parameter called alpha: the values of alpha are greater than or equal to zero but cannot equal one. The Renyi entropy ordering is related to the underlying probability distributions and allows more probable events to be weighted more heavily. As alpha approaches zero, the Rényi entropy increasingly weighs all possible events more equally, regardless of their probabilities. A higher alpha (a) weighs more probable events more heavily. The base used to calculate entropies is usually base 2 or Euler's number base e. If the base of the logarithm is 2, then the uncertainty is measured in bits. If it is the natural logarithm, then the unit is nats.

##### Rényi entropies

The Rényi entropy of order $\alpha$, where $\alpha \geq 0$  and $\alpha \neq 1$ , is defined as

$$
H_\alpha(X) = \frac{1}{1-\alpha}\log\Bigg(\sum_{i=1}^n p_i^\alpha\Bigg)
$$

Here, X is a discrete random variable with possible outcomes 1,2,...,n and corresponding probabilities $p_i \doteq \Pr(X=i) for i=1,\dots,n,$ and the logarithm is base 2.

#### Hartley entropy

The Hartley entropy (Gray, 1990) is the Rényi entropy with an alpha of zero.

the probabilities are nonzero, $H_0$ is the logarithm of the cardinality of X, sometimes called the Hartley entropy of X:

$$
H_0 (X) = \log n = \log |X|
$$

#### Shannon entropy

The Shannon entropy (Gray, 1990) is the Rényi entropy with an alpha of one. The Shannon entropy is a simple estimate of the expected value of the information contained in a message. It assumes independence and identically distributed random variables, which is a simplification when applied to word counts. In this sense it is analogous to naïve Bayes, in that it is very commonly used and thought to work well in spite of violating some assumptions upon which it is based.

The limiting value of $H_\alpha as \alpha \rightarrow 1$ is the Shannon entropy:

$$
H_1 (X) = - \sum_{i=1}^n p_i \log p_i.
$$

#### collision entropy

The collision entropy (Gray, 1990) is the Rényi entropy with an alpha of two and is sometimes just called "Rényi entropy," refers to the case $\alpha = 2$,

$$
H_2 (X) = - \log \sum_{i=1}^n p_i^2 = - \log P(X = Y)
$$

where $X$ and $Y$ are independent and identically distributed.

#### min-entropy

The min-entropy (Gray, 1990) is the Rényi entropy as the limit of alpha approaches infinity. The name min-entropy stems from the fact that it is the smallest entropy measure in the Rényi family of entropies. In the limit as $\alpha \rightarrow \infty$, the Rényi entropy $H_\alpha converges to the min-entropy H_\infty$:

$$
H_\infty(X) \doteq \min_i (-\log p_i) = -(\max_i \log p_i) = -\log \max_i p_i\,.
$$

Equivalently, the min-entropy $H_\infty(X)$ is the largest real number b such that all events occur with probability at most $2^{-b}$.

#### Kullback-Leibler divergence

[Kullback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) (Gray, 1990) is a non-symmetric measure of the difference between two probability distributions. The Kullback-Leibler measure goes by several names: relative entropy, discrimination information, Kullback-Leibler (KL) number, directed divergence, informational divergence, and cross entropy. Kullback-Leibler divergence is a measure of the difference between the observed entropy and its excepted entropy. We calculate the KL divergence by weighting one distribution (like an observed frequency distribution) by the log of probabilities of some other distribution D2. For discrete probability distributions P and Q, the Kullback–Leibler divergence of Q from P is defined to be

$$
D_{\mathrm{KL}}(P\|Q) = \sum_i P(i) \, \ln\frac{P(i)}{Q(i)}
$$

In words, it is the expectation of the logarithmic difference between the probabilities P and Q, where the expectation is taken using the probabilities P.

#### Mutual Information

[Mutual information](https://en.wikipedia.org/wiki/Mutual_information) (Gray, 1990) quantifies the mutual dependence of the two random variables. It is a measure of the “stickiness” between two items. It measures how much knowing one of these variables reduces uncertainty about the other. We can use mutual information to quantify the association between two tags. Mutual information (Equation 10) is given by:

the mutual information of two discrete random variables X and Y can be defined as:

$$
 I(X;Y) = \sum_{y \in Y} \sum_{x \in X}
                 p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
                              \right) }, \,\!
$$

where $p(x,y)$ is the joint probability distribution function of $X$ and $Y$, and $p(x)$ and $p(y)$ are the marginal probability distribution functions of $X$ and $Y$ respectively. In the case of continuous random variables, the summation is replaced by a definite double integral:

$$
 I(X;Y) = \int_Y \int_X
                 p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
                              \right) } \; dx \,dy,
$$

where $p(x,y)$ is now the joint probability density function of $X$ and $Y$, and $p(x$) and $p(y)$ are the marginal probability density functions of $X$ and $Y$ respectively.

## Zipf's law

[Zipf's law](https://en.wikipedia.org/wiki/Zipf%27s_law) is an empirical law formulated using mathematical statistics, refers to the fact that many types of data studied in the physical and social sciences can be approximated with a Zipfian distribution, one of a family of related discrete power law probability distributions. It states that while only a few words are used very often, many or most are used rarely,

$P_n \sim 1/n^a$

where $P_n$ is the frequency of a word ranked nth and the exponent a is almost 1. This means that the second item occurs approximately 1/2 as often as the first, and the third item 1/3 as often as the first, and so on. The is a so-called "power law" distributon.

The law is very common when looking a the distributions of words in all lagnuages. In fact, it is named after the American linguist [George Kingsley Zipf](https://en.wikipedia.org/wiki/George_Kingsley_Zipf) when he observed this law in 1935 in his academic studies of word frequency.

## Text-mining pipeline

Common Natural language processing tasks, such as tokenization,  stemming, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, word disamiguation, summarization, and coreference resolution are often done on every new text document before other types of analysis. As such it is common to write a "pipeline" (i.e. a sequence of scripts) that automatically perform these tasks.

*Tokenization*

[Tokenization](http://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html) is the task of chopping it up into pieces, called tokens. This is often done by throwing away certain characters, such as punctuation.

*Stemming*

[Stemming](https://en.wikipedia.org/wiki/Stemming) reduces words to their word stem, base or root form. For example, "argue", "argued", "argues", "arguing", and "argus" reduce to the stem "argu."  The can allow for "argue", "argued", "argues" to be counted as a single word.

*Sentence segmentation*

[sentence segmentation](http://www.monlp.com/2012/03/13/segmenting-words-and-sentences/) is the process plitting text into words and sentences. This is often done by looking for certain punctuation (full stop, question mark, exclamation mark, etc.).

*Part-of-speech tagging*

Given some text [part-of-speech tagging](https://en.wikipedia.org/wiki/Part-of-speech_tagging) determines the [part of speech](https://en.wikipedia.org/wiki/Part_of_speech) for each word. (e.g. a noun, verb, adjective. etc.)

*Named entity extraction*

[Named entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition) determines which items in the text map to proper nouns, such as people or places, or things.

*Chunking*

Chunking is also called shallow parsing. It is the identification short meaningful n-grams (like noun phrases, named entities, collocations, etc.).  In this processing of a tweet, it could refer to sperating the hashtags, the links and the tweet.

*Parsing*

[Parsing](https://en.wikipedia.org/wiki/Parsing) or syntactic analysis generates [https://en.wikipedia.org/wiki/Parsing] the parse tree (grammatical analysis) for each sentence or fragment.

*Word-sense disambiguation*

Many words have more than one meaning. [Word-sense disambiguation](https://en.wikipedia.org/wiki/Word-sense_disambiguation] selectx the meaning which makes the most sense in context.

*Coreference resolution*

Given a sentence or larger chunk of text, [coreference resolution](https://en.wikipedia.org/wiki/Coreference) determines which words ("mentions") refer to the same objects ("entities")

*Automatic summarization*

Automatic summarization produce a readable summary of a chunk of text. The summary need not be grammatical, for example, a [tag cloud](https://en.wikipedia.org/wiki/Tag_cloud) could be thought of as a visual summarization of some text.

Which elements are part of a Text-mining pipeline depend on the application.

## Small Talk and the Social Web

One particularly interesting aspect of the social web is the nature of text made available for public consumption. From the time of the Gutenberg printing press until the advent of Web 2.0, nearly all text presented in public was written by professionals. Whether it was a book, a business or government record, a sermon, a news or opinion article, a scientific paper, or an advertisement, it was written by a professional with the intent to communicate information and/or ideas. Not until the social web and Twitter did musings about what a noncelebrity ate for breakfast or whether someone likes naps was widely available for public consumption.

Musings about one’s foot fungus or a statement like “ Hahahahahaha!!!! You should have come to NKLA!!! So many beautiful pitties! And pittie lovers....”  are what we call small talk. Small talk is light, intimate banter, often understandable only by the authors’ close friends. A lot of the communication on the social web is small talk, even though it was very rare in public writing prior to the social web.

This free very unstructured nature of language on the social web should be taken into account when mining these data. It can add a considerable amount of noise and magnifies the importance of good semantic and lexcial resources.

## Ensemble methods

In statistics and machine learning, [ensemble methods](https://en.wikipedia.org/wiki/Ensemble_learning) use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms.

Ensemble methods groups a collection of base learners, with each learning according to its arget function. The ensemble then combines their outputs for a final predication. The basic idea is "two heads are better than one.
" (assuming those heads make independent decsisons). This is sometimes called “meta-learning”.

*Ensemble Learning*

The basic issue with ensemble learning is:
* How can you choose independent learners?
* How can you combine learners?

## Ensemble theory

Ensemble learning build many models and combines them. ensemble methods can be shown both theoretically and empirically to outperform single predictors on a wide range of tasks.

*When does this work?*

* Let p be the probability that a classifier makes an error
* Assume that classifier errors are independent
* The probability that k of the n classifiers make an error

$$
\binom n k  p^k(1-p)^{n-k}
$$

for k = 0, 1, 2, ..., n, where
$$
\binom n k =\frac{n!}{k!(n-k)!}
$$
is the binomial coefficient, hence the name of the distribution.

Therefore the probability that a majority vote classifier is in error is:

$$
 \sum_{k>n/2}^{n} \binom n k  p^k(1-p)^{n-k}
$$

### Majority Vote Model

In the majority vote model the ensemble chooses the class predicted by more than $n/2$ the $n$ classifiers. If no agreement then return an error.

### When is an Ensemble Better than a Single Learner?

*“No Free Lunch” Theorem*

No single algorithm wins all the time! By combing multiple independent decisions, each of which is at least more accurate than random guessing, random errors cancel each other out and correct decisions are reinforced. Empirically, ensembles tend to yield better results when there is a significant diversity among the models.

*Bias Problem*

The hypothesis space made available by a particular classification method does not  include the true hypothesis. The bias is error from erroneous assumptions in the learning algorithm.

*Variance Problem*

The hypothesis space is “too large” for the amount of training data – thus selected hypothesis may be inaccurate on unseen data. The variance is error from sensitivity to small fluctuations in the training set. High variance can cause overfitting: modeling the random noise in the training data, rather than the intended outputs.

Bias and variance usually work in opposition to one other: attempts to reduce the bias component will cause an increase in variance, and vice versa. Many ensemble methods, therefore, seek to promote diversity among the models they combine. Ensembles methods tend to decrease error by decreasing the variance in the results due to unstable learners, algorithms  whose output can change dramatically when the training data is slightly changed. However, these random errors may not cancel and be further reinfornced if the learners all have the same bias.

### Condorcet Jury Theorem

[Condorcet's jury theorem](https://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem) is a political science theorem about the relative probability of a given group of individuals arriving at a correct decision. The theorem was first expressed by the Marquis de Condorcet in his 1785 work Essay on the Application of Analysis to the Probability of Majority Decisions.

If each voter has an independent probability p of voting for the correct decision then the critical p for which majority voting works is a probability greater than or less than 1/2:

* If p is greater than 1/2 (each voter is more likely to vote correctly), then adding more voters increases the probability that the majority decision is correct. In the limit, the probability that the majority votes correctly approaches 1 as the number of voters increases.

* On the other hand, if p is less than 1/2 (each voter is more likely to vote incorrectly), then adding more voters makes things worse: the optimal jury consists of a single voter.

* It is critical that voter has an independent probability.

The probability of a correct majority decision P(n,p) (p close to 1/2), the gain by having n voters grows proportionally to \sqrt{n}.

$$
 P(n,p) = 1/2 + c_1 (p-1/2) + c_3 (p-1/2)^3 + O( (p-1/2)^5 )
$$
where
$$
 c_1 = {n  \choose { \lfloor n/2 \rfloor}} \frac{ \lfloor n/2 \rfloor +1} { 4^{\lfloor n/2 \rfloor}} = \sqrt{ \frac{2n+1}{\pi}} (1 + \frac{1}{16n^2} + O(n^{-3}) )
$$

### Wisdom of Crowds

The "Wisdom of Crowds" refers to situations in which human ensembles are demonstrably better than individual experts. This has often been observed in situations like:

* How many jelly beans are in the jar?
* Who Wants to be a Millionaire:  “Ask the audience”
* Accurately guessing the weight of an ox

Many examples are given in [James Surowiecki's](https://en.wikipedia.org/wiki/James_Surowiecki) book The [Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations](https://en.wikipedia.org/wiki/The_Wisdom_of_Crowds), published in 2004.

Surowiecki notes that Not all crowds (groups) are wise.  Charles Mackay's [Extraordinary Popular Delusions and the Madness of Crowds](https://en.wikipedia.org/wiki/Extraordinary_Popular_Delusions_and_the_Madness_of_Crowds), published in 1841, is a history of popular crowd delusions. Rather he states four key criteria separate wise crowds from irrational ones:

* Diversity of opinion - Each person should have private information even if it's just an eccentric interpretation of the known facts.
* Independence - People's opinions aren't determined by the opinions of those around them.
* Decentralization - People are able to specialize and draw on local knowledge.
* Aggregation	Some mechanism exists for turning private judgments into a collective decision.

For ensemble learners certainly the following are critical

* Independence
* Aggregation

and the following certainly helps

* Diversity of learners

It's not so clear that ensemble learners should specialize and draw on local knowledge; but the effect is related to diversity.

## Common types of ensembles

### Bootstrap aggregating (bagging)

Create ensembles by [“bootstrap aggregation”](https://en.wikipedia.org/wiki/Bootstrap_aggregating), i.e., repeatedly randomly re-sampling training data. Not that bagging uses the same learner so bias related to the method isn't addressed by this approach.

Bootstrap: draw n items from X with replacement

Bootstrap aggregating: combines random learners (often with voting, averaging or median) to create a predictor lesss efected by noise. Unstable and/or noisy algorithms often profit from bagging.

Bagging's usefulness depends on the stability of the base classifiers.  If small changes in the sample cause small changes in the base-level classifier, then the ensemble will not be much better than the base classifiers. It reduces variance and helps to avoid overfitting. It is often applied to decision tree methods (random forests) and nearest neighbor classifiers, but it can be used with any type of method.

### Boosting

Boosting involves incrementally building an ensemble by training each new model instance to emphasize the training instances that previous models mis-classified. In these sense it "learns." Unlike bagging, weights may change at the end of boosting round making certain learners more important than others. In some cases, boosting has been shown to yield better accuracy than bagging, but it also tends tp propgate bias from the overweighting winning predictor and is more likely to over-fit the training data. By far, the most common implementation of Boosting is Adaboost.

*Summary of Boosting and Bagging*

* Called “homogenous ensembles” becuase there is a single learner
* Bagging: Resample training data
* Boosting: Reweight training data

### AdaBoost

[AdaBoost](https://en.wikipedia.org/wiki/AdaBoost), short for "Adaptive Boosting." AdaBoost refers to a particular method of training a boosted classifier. A boost classifier is a classifier in the form

$$
F_T(x) = \sum_{t=1}^T f_t(x)\,\!
$$

where each $f_t$ is a weak learner that takes an object $x$ as input and returns a real valued result indicating the class of the object.

Each weak learner produces an output, hypothesis $h(x_i)$, for each sample in the training set. At each iteration t, a weak learner is selected and assigned a coefficient $\alpha_t$ such that the sum training error $E_t$ of the resulting t-stage boost classifier training error is minimized.

$E_t = \sum_i E[F_{t-1}(x_i) + \alpha_t h(x_i)]$
Here $F_{t-1}(x)$ is the boosted classifier that has been built up to the previous stage of training, $E(F)$ is some error function and $f_t(x) = \alpha_t h(x)$ is the weak learner that is being considered for addition to the final classifier.

AdaBoost can be thought of as a form of gradient descent along the error gradient.

### Bayesian parameter averaging

Bayesian classifiers are determined by the parameters of the models. Bayesian parameter averagingis an ensemble of all the hypotheses in the hypothesis space. The idea is that the bias of the modelds does not change, but variance decreases by the number of independent models.

### Bucket of models

A "bucket of models" is an ensemble in which a model selection algorithm is used to choose the best model for each problem. When tested with only one problem, a bucket of models can produce no better results than the best model in the set, but when evaluated across many problems, it will typically produce much better results, on average, than any model in the set.

```
For each model m in the bucket:
  Do c times: (where 'c' is some constant)
    Randomly divide the training dataset into two datasets: A, and B.
    Train m with A
    Test m with B
Select the model that obtains the highest average score
```
This idea can be summed up as, "try them all with the training set, and pick the one that works best".

### Stacking

Stacking (sometimes called stacked generalization) involves training a learning algorithm to combine the predictions of several other learning algorithms.

Unlike a bucket of models it isn't a "winner take all" approach. Unlike “homogenous ensembles” like bagging and boosting here we are creating “heterogenous ensembles” and various preictors.

### Random Forests

[Random forests](https://en.wikipedia.org/wiki/Random_forest) involve bagging multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees.

```
For i = 1 to T,
  Take a bootstrap sample (bag)
  Grow a random decision tree  T_i
  At each node choose a feature from one of n features (n < total number of features)
  Grow a full tree (do not prune)

Final classification is done by majority vote across the T random trees.
```

## Developing Ensemble Models?

The main challenge is not to obtain highly accurate base models, very weak predictors with an accuracy of greater than 50% can be used, but rather to obtain base models which make different kinds of errors.

When one resamples or rewights training data, as one does with bagging amd boosting it helps with noisy data but not the predictor itself.

To generate “heterogenous ensembles” independence between the base classifiers needs to be assessed. Independence between classifiers is often assed by measuring the degree of overlap in misclassifying training examples. Less overlap means more independence between models. The key of designing ensembles is diversity and not necessarily high accuracy of the base classifiers: Members of the ensemble should vary in the examples they misclassify.

## Probability and Statistics

Probability is a measure of the likelihood of a random phenomenon or chance behavior.  Probability describes the long-term proportion with which a certain outcome will occur in situations with short-term uncertainty.

### The Axioms of Probability

##### First axiom - The probability of an event is a non-negative real number:
$$
P(E)\in\mathbb{R}, P(E)\geq 0 \qquad \forall E\in F
$$
where $F$ is the event space

##### Second axiom -  unit measure:

The probability that some elementary event in the entire sample space will occur is 1.

$$
P(\Omega) = 1.
$$

##### Third axiom - the assumption of $\sigma$-additivity:

Any countable sequence of disjoint (synonymous with mutually exclusive) events $E_1, E_2, ...$ satisfies

$$
P\left(\bigcup_{i = 1}^\infty E_i\right) = \sum_{i=1}^\infty P(E_i).
$$

The total probability of all possible event always sums to 1.

### Consequences of these axioms
The probability of the empty set:
$$
P(\varnothing)=0.
$$

Monotonicity
$$
\quad\text{if}\quad A\subseteq B\quad\text{then}\quad P(A)\leq P(B).
$$

The numeric bound between 0 and 1:

$$
0\leq P(E)\leq 1\qquad \forall E\in F.
$$

*Probability is expressed in numbers between 0 and 1.*

Probabilty of a certain event is 1:

$$
P(True) = 1
$$

Probability = 1 means it always happens.

Probabilty of an impossible event is 0:

$$
P(False) = 0
$$

Probability = 0 means the event never happens.

Probabilty of A or B:

$$
P(A \quad or \quad B) = P(A) + P(A) - P(A \quad and \quad B)
$$

or

$$
P(A \cup B) = P(A) + P(A) - P(A \cap B)
$$

Probabilty of not A:

$$
P(not  \quad A) = 1- P(A)
$$

## Conditional Probability

In probability theory, a [conditional probability](https://en.wikipedia.org/wiki/Conditional_probability) measures the probability of an event given that another event has occurred. That is,  "the conditional probability of A given B."

 the conditional probability of A given B is defined as the quotient of the probability of the joint of events A and B, and the probability of B:

$$
P(A|B) = \frac{P(A \cap B)}{P(B)}
$$

This may be visualized using a Venn diagram.

*$P(A \cap B)$*

### Corollary of Conditional Probability is The Chain Rule

If we multiply both sides by $P(B)$ then

$$
P(A|B) = \frac{P(A \cap B)}{P(B)}
$$

becomes

$$
P(A|B) P(B) = P(A \cap B)
$$

### Statistical independence

Events A and B are defined to be statistically independent if:

$$
\begin{align}
             P(A \cap B) &= P(A) P(B) \\
  \Leftrightarrow P(A|B) &= P(A) \\
  \Leftrightarrow P(B|A) &= P(B)
\end{align}
$$

That is, the occurrence of A does not affect the probability of B, and vice versa

Probabilty of A or B for independent events $P(A and B) is 0$:

$$
P(A or B) = P(A) + P(A)
$$

## Bayes Rule

[Bayes' theorem](https://en.wikipedia.org/wiki/Bayes%27_theorem) (alternatively Bayes' law or Bayes' rule) describes the probability of an event, given prior events. That is, a conditional probability.

$$
P(A|B) = \frac{P(A)\, P(B | A)}{P(B)},
$$

where A and B are events.

* P(A) and P(B) are the independent probabilities of A and B.
* P(A | B), a conditional probability, is the probability of observing event A given that B is true.
* P(B | A), is the probability of observing event B given that A is true.

## Bayesian inference

[Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference) is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as evidence. Bayesian inference derives the posterior probability as a consequence of two antecedents, a prior probability and a "likelihood function" derived from a statistical model for the observed data.

Bayesian inference computes the posterior probability according to Bayes' theorem:

$$
P(H\mid E) = \frac{P(E\mid H) \cdot P(H)}{P(E)}
$$

where,

$P(H\mid E)$ the posterior probability, denotes a conditional probability of $\textstyle H$ (the hypothesis) whose probability may be affected by the evidence $\textstyle E$.

$\textstyle P(H)$, the prior probability, is an estimate of the probability that a hypothesis is true, before observing the current evidence.

$\textstyle P(E\mid H)$ is the probability of observing $\textstyle E$ given $\textstyle H$. It indicates the compatibility of the evidence with the given hypothesis.

$\textstyle P(E)$ is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible hypotheses being considered.

Note that Bayes' rule can also be written as follows:
$$
P(H\mid E) = \frac{P(E\mid H)}{P(E)} \cdot P(H)
$$
where the factor $\textstyle \frac{P(E\mid H)}{P(E)}$ represents the impact of $E$ on the probability of $H$.

## Bayesian probability example

Suppose a certain disease has an incidence rate of 0.01% (that is, it afflicts 0.01% of the population).  A test has been devised to detect this disease.  The test does not produce false negatives (that is, anyone who has the disease will test positive for it), but the false positive rate is 1% (that is, about 1% of people who take the test will test positive, even though they do not have the disease).  Suppose a randomly selected person takes the test and tests positive.  What is the probability that this person actually has the disease?

Bayes theorem would ask the question, what is the probability of disease given a postive result, or $P(disease\mid positive))$.

What do we know?

$P(positive\mid disease)=1$ (i.e. The test does not produce false negatives.)
$P(disease)=0.0001$ (i.e.  1/10,000 have the disease)
$P(positive\mid no disease)=0.01$ (i.e. he false positive rate is 1%. This means 1% of people who take the test will test positive, even though they do not have the disease)

Bayes’ Theorem

$$
P(A|B) = \frac{P(A)\, P(B | A)}{P(B)},
$$

which can be rewritten as

$$
P(A|B) = \frac{P(A)\, P(B | A)}{P(A)P(B|A)+P(\bar{A})P(B|\bar{A})},
$$

which in our example is
$$
P(disease|positive) = \frac{P(disease)\, P(positive | disease)}{P(disease)P(positive|disease)+P(no \quad  disease)P(positive|no \quad disease)},
$$

plugging in the numbers gives

$$
P(disease|positive)= \frac{(0.0001)\, (1)}{(0.0001)(1)+(0.9999)(0.01)}, \approx 0.01
$$

So even though the test is 99% accurate, of all people who test positive, over 99% do not have the disease.

## Bayesians versus Frequentists

[Frequentist inference](https://en.wikipedia.org/wiki/Frequentist_inference) or frequentist statistics is a scheme for making statistical inference based on the frequency or proportion of the data. This effectively requires that conclusions should only be drawn with a set of repetitions.

Frequentists will only generate statistical inference given a large enough set of repetitions. In contrast, a Bayesian approach to inference does allow probabilities to be associated with unknown parameters.

*Count Von Count*
- from https://en.wikipedia.org/wiki/File:Count_von_Count_kneeling.png

While "probabilities" are involved in both approaches to inference, frequentist probability is essentially equivelent to counting. The Bayesian approach allows these estimates of probabilities to be based upon counting but also allows for subjective estimates (i.e. guesses) of prior probabilities.

Bayesian probability, also called evidential probability, or subjectivist probability, can be assigned to any statement whatsoever, even when no random process is involved. Evidential probabilities are considered to be degrees of belief, and a Bayesian can even use an un-informative prior (also called a non-informative or Jeffreys prior).

In Bayesian probability, the [Jeffreys prior](https://en.wikipedia.org/wiki/Jeffreys_prior), named after Sir Harold Jeffreys, is a non-informative (objective) prior distribution for a parameter space.  The crucial idea behind the Jeffreys prior is the Jeffreys posterior. This posterior aims to reflect as best as possible the information about the parameters brought by the data, in effect  "representing ignorance" about the prior. This is sometimes called the "principle of indifference." Jeffreys prior is proportional to the square root of the determinant of the Fisher information:

$$
p\left(\vec\theta\right) \propto \sqrt{\det \mathcal{I}\left(\vec\theta\right)}.\,
$$

It has the key feature that it is invariant under reparameterization of the parameter vector $\vec\theta.$

At its essence the Bayesian can be vague or subjective about an inital guess at a prior probability. and the the posterior probability be updated data point by data point. A Bayesian defines a "probability" in the same way that many non-statisticians do - namely an indication of the plausibility or belief of a proposition.

A Frequentist is someone that believes probabilities represent long run frequencies with which events occur; he or she will have a model (e.g. Guassian, uniform, etc.) of how the sample popluation was generated. The observed counts are considered a random sample the estimate the true parameters of the model.

It is important to note that most Frequentist methods have a Bayesian equivalent (that is, they give the same results) when there are enough repeated trails. That is, they converge the the same result given enough data.

## Genetic Algorithms

A genetic algorithm (GA) is a search heuristic that mimics the process of natural selection. This heuristic is often used to generate useful solutions to optimization and search problems.

*Genetic algorithm (GA)*

### Genetic Algorithm Pseudocode

Create an initial population, typically random

While the best candidate so far is not a solution:
   Create new population using crossover and mutation.
   Evaluate the fitness of each candidate in the population.
   Replace/delete least-fit population

Return the best candidate found

### Genetic Algorithm Basic components

* Candidate representation
    + Important to choose this well the form can effect the solution.
    + The typical candidate representation is a binary string.
* successor functions.
    + Mutation, crossover
    + Mutation - Given a candidate, return a slightly different candidate.
    + Crossover - Given two candidates, produce one that has elements of each.
* Fitness function
    + The fitness function quantitates estimates how close a candidate is to being a solution.
* Solution test
    + Check whether the candidate is a solution.
* Some parameters
    + Population size
    + Generation limit

### Pros and Cons

*Pros*
*  Fast (and low memory)
*  Finding a candidate representation and fitness function are the bulk of the work.

*Cons*
* Randomized (not guaranteed optimal or complete).
* Can get stuck on local maxima (crossover is intended to help get out of local maxima)

## Simulated Annealing

A Simulated Annealing (SA) is a  probabilistic search heuristic that mimics the process of cooling in thermodynamic systems. This heuristic is often used to generate useful solutions to optimization and search problems.  The method is an adaptation of the [Metropolis–Hastings algorithm](https://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_algorithm), a Monte Carlo method to generate sample states of a thermodynamic system, invented by M.N. Rosenbluth and published in a paper by N. Metropolis et al. in 1953.

Basically simulated annealing perturbs the current solution  and then checks to see whether the new solution is good or not. If its an improvement it will accept it and if the new solution is worse it may accept it with a probability inversely proportional to how much worse the new solution changes the current one. Compared to pure gradient descent the main difference is that SA allows "uphill" steps. Simulated annealing also differs from gradient descent in that a move is selected at random.

*Gradient Descent*

### Metropolis–Hastings algorithm

[Metropolis algorithm](https://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_algorithm) (symmetric proposal distribution)
Let f(x) be a function that is proportional to the desired probability distribution P(x)

This algorithm proceeds by randomly attempting to move about the sample space, sometimes accepting the moves and sometimes remaining in place.

Perturb (randomly) the current state to a new state.
$\Delta E$ is the difference in energy between current and new state.
If $\Delta E < 0$ (new state is lower), accept new state as current state
If  $\Delta E > 0$  accept new state with probability inversely proportional to the increase in system energy. Traditionally the change in [Gibbs free energy](https://en.wikipedia.org/wiki/Gibbs_free_energy) is used for thermodynamic free energy systems.

This can be run for a fixed number of iterations or if the overall system energy can measured then it can be run until the overall system energy settles.

### Simulated Annealing Pseudocode

Simulated Annealing uses the Metropolis–Hastings algorithm with a temperature parameter $T$ that effects the acceptance probability of an "uphill" transition. At higher"temperatures" accpeting "uphill" transitions is more probable. The algorithm starts initially with $T$ set to a high value , and then it is decreased at each step following some annealing schedule—which is often specified by the user, but must end with $T=0$. At $T=0$ there is no chance of accpeting "uphill" transitions and so it becomes gradient descent.

At a fixed temperature T:
Perturb (randomly) the current state to a new state.
$\Delta E$ is the difference in energy between current and new state.
If $\Delta E < 0$ (new state is lower), accept new state as current state
If  $\Delta E > 0$  accept new state with probability inversely proportional to the increase in system energy as a function of T.

Eventually the systems evolves into thermal equilibrium at temperature T ; then the formula mentioned before holds When equilibrium is reached, temperature T can be lowered and the process can be repeated.

## Travelling Salesman Problem

The travelling salesman problem (TSP) asks the following question: Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once and returns to the origin city? It is an NP-hard problem in combinatorial optimization, important in operations research and theoretical computer science.

## Self-Organizing Maps (SOM)

A [self-organizing map (SOM)](https://en.wikipedia.org/wiki/Self-organizing_map) or self-organizing feature map (SOFM) is an artificial neural network (ANN) that is trained using unsupervised learning to produce a low-dimensional (typically two-dimensional), discretized representation of the input space of the training samples, called a map. This makes SOMs useful for visualizing low-dimensional views of high-dimensional data. In this sense, SOMs are similar to multidimensional scaling. Self-organizing maps with a small number of nodes behave in a way that is similar to K-means, larger self-organizing maps rearrange data in a way that is fundamentally topological in character.

*SOM Algorithm*

*SOM Training*

### SOM neighborhood topology

*Square Grid Lattice SOM]*

*Hexoganal Honeycomb Lattice SOM]*

The grids have an $x$ dimension and $y$ dimension. The topology of the grid is rectangular or hexagonal.

### SOM neighborhood functions

While the neighborhood function $\theta(u, v, s)$ usually depends on the lattice distance between the BMU (neuron u) and neuron v. Other functions such as a [Gaussian function](https://en.wikipedia.org/wiki/Gaussian_function) or a radius distance centered around the neuron are also common choices.

### SOM Algorithm

A. Randomize the map's nodes' weight vectors. The weights of the neurons are initialized either to small random values or sampled evenly from the subspace spanned by the two largest principal component eigenvectors.

B.  Grab an input vector $\mathbf{D(t)}$. $t$ is the index of the target input data vector in the input data set $\mathbf{D}$ Each data point, $\mathbf{D(t)}$, needs to be mapped to a "neuron."

C.  Traverse each node in the map. Use the Euclidean distance or another similarity metric between the input vector $\mathbf{D(t)}$ and the map's node's weight vector. The node that produces the smallest distance (i.e. the best matching unit, BMU) is associated with that neuron.

D. Update the nodes in the neighborhood of the BMU (including the BMU itself) by pulling them closer to the input vector. This is analogous to updating the new means to be the centroids of the observations in the new clusters the means in [k-means clustering](https://en.wikipedia.org/wiki/K-means_clustering). That is,

$$
Wv(s + 1) = Wv(s) + \theta(u, v, s) 	\alpha(s)(D(t) - Wv(s))
$$

Where

* $s$ is the current iteration
* $\lambda$ is the iteration limit. Sometimes this is called the maximum number of 'epochs'
* $v$ is the index of the node in the map
* $\mathbf{W_v}$ is the current weight vector of node v
* $u$ is the index of the best matching unit (BMU) in the map
* $\Theta (u, v, s)$ is a restraint due to distance from BMU, usually called the neighborhood function,
* $\alpha (s)$ is a learning restraint due to iteration progress. Sometimes this is called the learning rate.

E Increase $s$ and repeat from step B while $s < \lambda$

Note that the neighborhood function $\Theta (u, v, s)$  depends on the lattice distance between the BMU (neuron u) and neuron v. In the simplest form it is 1 for all neurons close enough to BMU and 0 for others, but a Gaussian function is a common choice, too. Regardless of the functional form, the neighborhood function shrinks with time.

A variant algorithm:

A. Randomize the map's nodes' weight vectors
B. Traverse each input vector in the input data set
C. Traverse each node in the map
D.  Use the Euclidean distance formula to find the similarity between the input vector and the map's node's weight vector
E.  Track the node that produces the smallest distance (this node is the best matching unit, BMU)
Update the nodes in the neighborhood of the BMU (including the BMU itself) by pulling them closer to the input vector
$Wv(s + 1) = Wv(s) + \theta(u, v, s) \alpha(s)(D(t) - Wv(s))$
F. Increase $s$ and repeat from step B while $s < \lambda$

The kind of training is called *[competitive learning](https://en.wikipedia.org/wiki/Competitive_learning)* n which nodes compete for the right to respond to a subset of the input data.

### SOM Usage

Using Self-Organising Maps are typcially done as follows:

Select the size and type of the map. The shape can be hexagonal or square. Note that hexagonal grid has six immediate neighbors whereas a square usually has four. This topology determines the number of "neurons."

The algorithm then

A. Initializes all node weight vectors.
B. Chooses a random data point from the training data to present to the SOM.
C. Finds the "Best Matching Unit" (BMU) in the map.
Determine the nodes within the "neighborhood" of the BMU.
D. The size of the neighborhood decreases with each iteration.
E. Adjust weights of nodes (neurons) in the BMU neighborhood  towards a chosen datapoint.
- The learning rate decreases with each iteration.
- The magnitude of the adjustment is proportional to the proximity of the node to the BMU.

Repeat Steps B-E for a fixed number of iterations or until convergence.

### SOM Pros and Cons

Pros:

* Intuitive method.
* Simple algorithm.
* New data points can be mapped to trained model for predictive purposes.

Cons:

* Requires clean, numeric data.

## SOM Visualization

The Self-Organizing Maps projects high-dimensional data onto a two-dimensional map. The projection preserves the topology of the data so that similar data items will be mapped to nearby locations on the map. As such one of its great strengths is high-dimensional data visualization, akin to [multidimensional scaling (MDS)](https://en.wikipedia.org/wiki/Multidimensional_scaling).

### SOM Cluster Maps

*SOM Clusters*
- from [Self-Organising Maps for Customer Segmentation using R](http://www.r-bloggers.com/self-organising-maps-for-customer-segmentation-using-r/) via @rbloggers

*SOM Clusters with Labels*
- from [Self-Organising Maps for Customer Segmentation using R](http://www.r-bloggers.com/self-organising-maps-for-customer-segmentation-using-r/) via @rbloggers

*SOM Clusters on Map*
- from [Self-Organising Maps for Customer Segmentation using R](http://www.r-bloggers.com/self-organising-maps-for-customer-segmentation-using-r/) via @rbloggers
### SOM Heatmaps

A [heat map](https://en.wikipedia.org/wiki/Heat_map) is a graphical representation of data where a gradient of values are represented as colors.

*SOM Heatmap*
- from [Self-Organising Maps for Customer Segmentation using R](http://www.r-bloggers.com/self-organising-maps-for-customer-segmentation-using-r/) via @rbloggers

## Data pipelines

Data Pipeline is an embedded data processing engine. The engine runs inside your applications, APIs, and jobs to filter, transform, and migrate data in an automated way.

## SQL

SQL (pronounced "ess-que-el") stands for Structured Query Language. SQL is used to communicate with a [relational database](https://en.wikipedia.org/wiki/Relational_database).  A relational database is a digital database whose organization is based on the relational model of data, as proposed by E. F. Codd in 1970. The various software systems used to maintain relational databases are known as a relational database management system (RDBMS). Virtually all relational database systems use SQL (Structured Query Language) as the language for querying and maintaining the database.

## NoSQL

A [NoSQL](https://en.wikipedia.org/wiki/NoSQL) (originally referring to "non SQL" or "non relational") database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases.

Motivations for this approach include: simplicity of design, spped, simpler "horizontal" scaling to clusters of machines (which is a problem for relational databases), and finer control over availability. The data structures used by NoSQL databases (e.g. key-value, wide column, graph, or document) are different from those used by default in relational databases, making some operations faster in NoSQL. The particular suitability of a given NoSQL database depends on the problem it must solve.

## MapReduce

[MapReduce](https://en.wikipedia.org/wiki/MapReduce) is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster.[

A MapReduce program is composed of a Map() procedure (method) that performs filtering and sorting (such as sorting students by first name into queues, one queue for each name) and a Reduce() method that performs a summary operation (such as counting the number of students in each queue, yielding name frequencies).

The "MapReduce System" (also called "infrastructure" or "framework") orchestrates the processing by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance.

## Neural Networks

I highly recommend _Deep Learning - Adaptive Computation and Machine Learning series by Ian Goodfellow, Yoshua Bengio,
and Aaron Courville_ [https://github.com/HFTrader/DeepLearningBook](
https://github.com/HFTrader/DeepLearningBook)

and

[6.S191: Introduction to Deep Learning](http://introtodeeplearning.com/)

### Neural Networks

[Artificial neural networks (ANNs)](https://en.wikipedia.org/wiki/Artificial_neural_network) or connectionist systems are computing systems inspired by the biological neural networks that constitute animal brains.

**Neural networks** consist of:

_Input layer_

Neurons that maps the input to a neural network.

_Output layer_

Neurons that maps the output to a neural network. Often converts to output to a probability.

*	Softmax (If you want the outputs of a network to be interpretable as posterior probabilities.)
*	Cros

_Activation functions_

The [activation function](https://en.wikipedia.org/wiki/Activation_function) of a node defines the output of that node given an input or set of inputs.

*	Rectified linear unit (ReLU)
*	TanH
*	Leaky rectified linear unit (Leaky ReLU)
*	Parameteric rectified linear unit (PReLU)
*	Randomized leaky rectified linear unit (RReLU)
*	Exponential linear unit (ELU)
*	Scaled exponential linear unit (SELU)
*	S*shaped rectified linear activation unit (SReLU)
*	Identity
*	Binary step
*	Logistic
*	ArcTan
*	Softsign
*	Adaptive piecewise linear (APL)
*	SoftPlus
*	SoftExponential
*	Sinusoid
*	Sinc
*	Gaussian

_Comparison of activation functions_

Some desirable properties in an activation function include:
* Nonlinear – When the activation function is non-linear, then a two-layer neural network can be proven to be a universal function approximator. The identity activation function does not satisfy this property. When multiple layers use the identity activation function, the entire network is equivalent to a single-layer model.
* Continuously differentiable – This property is necessary for enabling gradient-based optimization methods. The binary step activation function is not differentiable at 0, and it differentiates to 0 for all other values, so gradient-based methods can make no progress with it.
* Range – When the range of the activation function is finite, gradient-based training methods tend to be more stable, because pattern presentations significantly affect only limited weights. When the range is infinite, training is generally more efficient because pattern presentations significantly affect most of the weights. In the latter case, smaller learning rates are typically necessary.
* Monotonic – When the activation function is monotonic, the error surface associated with a single-layer model is guaranteed to be convex.
* Smooth Functions with a Monotonic derivative – These have been shown to generalize better in some cases. The argument for these properties suggests that such activation functions are more consistent with Occam's razor.
* Approximates identity near the origin – When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.

_Objective function (cost function or loss function)_

The objective function estimates the error. (Is minimized during training)

*	Quadratic cost (mean*square error)
*	Cross*Entropy

_Back-Propagation_

[Backpropagation](https://en.wikipedia.org/wiki/Backpropagation) is a method used in artificial neural networks to calculate the error contribution of each neuron after a batch of data. This is used by an enveloping optimization algorithm to adjust the weight of each neuron, completing the learning process for that case.

Technically it calculates the gradient of the loss function. It is commonly used in the gradient descent optimization algorithm. It is also called backward propagation of errors, because the error is calculated at the output and distributed back through the network layers.

_Epochs_

One epoch = one forward pass and one backward pass.

*	number of epochs

_Gradient estimation_

*	stochastic gradient descent
*	Adagrad
*	RMSProp
*	ADAM
*	NAG
*	Adadelta
*	Momentum

_Batch size_

batch size = the number of training examples in one forward/backward pass. It's much faster to estimate the gradient on a sample than to calculate it over the entire data set.

_Batch normalization_

Normalization (shifting inputs to zero-mean and unit variance) is often used as a pre-processing step to make the data comparable across features.

Because a multiplies across the weights over each layer this can change the distribution. This is called _internal covariate shift._ As the data flows through a deep network, the weights and parameters adjust those values, sometimes making the data too big or too small again.

Internal covariate shift refers to the change in the input distribution to a learning system. In the case of deep networks, the input to each layer is affected by parameters in all the input layers.

Rather than just performing normalization once in the beginning, you're doing it through the process.

_Regularization_

*	L1
*	L2
*	Drop out

### Shallow Neural Networks

_Shallow Neural Networks_ refers to neural networks with few hidden layers.

### Deep Learning

_Deep Learning_

Deep feedforward networks feedforward , also often called neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models. The goal of a feedforward network is to approximate some function $f(x)$.

Earlier challenges in training deep neural networks were successfully addressed with methods such as unsupervised pre-training, while available computing power increased through the use of GPUs and distributed computing. Neural networks were deployed on a large scale, particularly in image and visual recognition problems. This became known as "deep learning", although deep learning is not strictly synonymous with deep neural networks it often refers to th neural networks with many hidden layers.

### Convolutional Networks

A convolutional networks (CNN) consists of an input and an output layer, as well as multiple hidden layers. The hidden layers are either convolutional, pooling or fully connected.

Convolutional layers apply a convolution operation to the input, passing the result to the next layer. The convolution emulates the response of an individual neuron to visual stimuli.

Each convolutional neuron processes data only for its receptive field. Tiling allows CNNs to tolerate translation of the input image (eg. translation, rotation, perspective distortion).

Although fully connected feedforward neural networks can be used to learn features as well as classify data, it is not practical to apply this architecture to images. A very high number of neurons would be necessary even in a shallow architecture (opposite of deep). The convolution operation brings a solution to this problem as it reduces the number of free parameters, allowing the network to be deeper with many fewer parameters. In other words, it resolves the vanishing or exploding problems in training traditional multi-layer neural networks with many layers by using backpropagation.

### Sequence Modeling: Recursive neural networks

[Recursive neural networks (RNNs)](https://en.wikipedia.org/wiki/Recursive_neural_network) is a kind of deep neural network created by applying the same set of weights recursively over a structure, to produce a structured prediction over variable-size input structures, or a scalar prediction on it, by traversing a given structure in topological order. RNNs have been successful for instance in learning sequence and tree structures in natural language processing, mainly phrase and sentence continuous representations based on word embedding. RNNs have first been introduced to learn distributed representations of structure, such as logical terms.

### Autoencoders

An autoencoder, autoassociator or Diabolo network is an artificial neural network used for unsupervised learning of efficient codings. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction.

Autoencoder’s though are difficult to interpret of the representation of semantics and aren’t really a generative model.

Autoencoder
i.	encoder and dencoder
ii.	hidden layers
iii.	bottleneck layer - forces network to learn a compressed latent representation
iv.	reconstruction loss  - forces hidden layer to represent info about the input
)

### Variational autoencoders (VAEs)

Variational autoencoders are a stochastic variational extension of autoencoders  that allow for a probabilistic representation of the data and amortized inference.

In a VAE, the encoder becomes a variational inference network that maps the data to the a distribution for the hidden variables, and the decoder becomes a generative network that maps the latent variables back to the data. In just a couple of years, Variational Autoencoders (VAEs) have emerged as one of the most popular approaches to unsupervised learning of complicated distributions.

Variational Autoencoders (VAEs) have some very desirable properties:

i.	Clear semantics as a generative model
ii.	Extension of autoencoders that allow sampling and estimating probabilities.
iii.	This creates a kind of implicit generative model it creates a latent representation thru its probabilities
iv.	"Latent variables" has a fixed prior distribution
v.	Probabilistic encoder and dencoder
vi.	Probabilistic representation of the data

Variational autoencoders have a clear advantage over autoencoders in that the probabilistic representation of the data is a form of representation learning.

_Autoregresstive variational autoencoders_

Standard VAE’s have some issues:
i.	They do not encode what is not useful for them to decode.  This means they don't capture fine details. Subtle features (e.g. small nodules that may be important clinically but don’t represent much of the image space may be missed.)

Autoregresstive VAE’s use autoregresstive networks in the encoder and the dencoder to capture more local info in the estimation of the densities.

### Restricted Boltzmann machines (RBMs)

Restricted Boltzmann machines (RBM)s are a very simple model, just a fully connected bipartite graph with an input layer and a hidden layer. The cost function minimizes an energy function by re-weighting to minimize the difference between the input layer and the hidden layer.

Like VAE’s restricted Boltzmann machines can be thought as a form of representation learning.

Restricted Boltzmann machines:
i.	Clear semantics as a generative model
ii.	Simple, just a fully connected bipartite graph with an input layer and a hidden layer.
iii.	Minimizes an energy function (re-weighting to minimize the difference between the input layer and the hidden layer).
iv.	Energies can easily be converted to probabilities.
v.	Creates a latent representation thru its probabilities.

### Deep Generative Models

Deep Generative Models collect a large amount of data in some domain (e.g., think millions of images, sentences, or sounds, etc.) and then train a model to generate data like it.

Last update September 5, 2017

Week_1/NBB_Big Data and Data Science.ipynb