# Topic modeling without Theta distribution

Based on unpublished (yet) research by Ilya Irhin, Victor Bulatov, Konstantin Vorontsov

One of the interpretations of topic modeling is this:

\begin{equation}
L(\Phi, \Theta) + R(\Phi, \Theta) \to \max_{\Phi, \Theta},
\end{equation}

Once the inference is directly formulated as an optimization problem, one can ask: is it possible to reduce the number f inferred parameters?

In the practice, word-topic matrix $(\Phi)$ has a larger importance than document-topic matrix $(\Theta)$; so we can replace the original optimization problem with a different one:

\begin{equation}
L(\Phi, f(\Phi) ) + R(\Phi, f(\Phi) ) \to \max_{\Phi},
\end{equation}

Some math later, it turns out that the "formally correct" optimization algorithm for this problem is different from conventional EM algorithm, and has some surprising advantages. Technically speaking, these formulas require a different iteration process (which is implemented here: https://github.com/ilirhin/python_artm), but the formulas look similar enough to a custom $\Phi$ regularizer (if one sets `num_document_passes=1` in BigARTM)

This notebook serves as a proof-of-concept for this idea.

```python
import sys
sys.path.insert(0, '../..')
```

```python
import numpy as np
import pandas as pd
import requests
import scipy.sparse

from numba import jit
from numpy import random
from matplotlib import pyplot as plt

import artm

from topicnet.cooking_machine.dataset import Dataset
from topicnet.cooking_machine.models import (
    BaseScore as BaseTopicNetScore,
    TopicModel
)
from topicnet.cooking_machine.models.base_regularizer import BaseRegularizer
from topicnet.cooking_machine.models.topic_prior_regularizer import TopicPriorRegularizer
from topicnet.cooking_machine.models.thetaless_regularizer import (
    ThetalessRegularizer,
    dataset2sparse_matrix,
)

from topicnet.viewers.top_tokens_viewer import TopTokensViewer
from topicnet.viewers.top_documents_viewer import TopDocumentsViewer
```

```python
%load_ext autoreload
%autoreload 2
```

```python
dataset = Dataset(
    "/data/datasets/20_News_dataset/20NG_BOW.csv",
    internals_folder_path="./20NG_internals",
)

dataset.get_possible_modalities()
```

    {'@footer', '@header', '@word'}

```python
MAIN_MODALITY = "@word"
```

## Preparing the coherence score

### Top-token coherence (co-occurrences are calculated using the same corpus)

```python
def calc_doc_occurrences(dataset, modality):
    """
    :param n_dw_matrix: sparse document-word matrix, shape is D x W
    :return: sparse matrix of co-occurrences

    doc_occurrences[w1, w2] = the number of the documents
    where there are w1 and w2
    """
    n_dw_matrix = dataset2sparse_matrix(dataset, modality, modalities_to_use=[modality])
    matrix = (scipy.sparse.csc_matrix(n_dw_matrix) > 0).astype(int)
    co_occurrences = matrix.T * matrix

    return co_occurrences.diagonal(), co_occurrences

def create_pmi_top_function(
    doc_occurrences, doc_co_occurrences,
    documents_number, top_sizes,
    co_occurrences_smooth=1.
):
    """
    :param doc_occurrences: array of doc occurrences of words
    :param doc_co_occurrences: sparse matrix of doc co-occurrences of words
    :param documents_number: number of the documents
    :param top_sizes: list of top values to calculate top-pmi for
    :param co_occurrences_smooth: constant to smooth co-occurrences in log
    :return: function which takes phi and theta and returns
    pair of two arrays: pmi-s of the tops and ppmi-s of the tops

    pmi[i] - pmi(top of size top_sizes[i])
    ppmi[i] - ppmi(top of size top_sizes[i])

    pmi(words) = sum_{u in words, v in words, u != v}
    log(
        (doc_co_occurrences[u, v] * documents_number + co_occurrences_smooth)
        / doc_occurrences[u] / doc_occurrences[v]
    )

    ppmi(words) = sum_{u in words, v in words, u != v}
    max(log(
        (doc_co_occurrences[u, v] * documents_number + co_occurrences_smooth)
        / doc_occurrences[u] / doc_occurrences[v]
    ), 0)

    """
    def func(phi):
        T, W = phi.shape
        max_top_size = max(top_sizes)
        pmi, ppmi = np.zeros(max_top_size), np.zeros(max_top_size)
        tops = np.argpartition(phi, -max_top_size, axis=1)[:, -max_top_size:]

        for t in range(T):
            top = sorted(tops[t], key=lambda w: - phi[t, w])
            co_occurrences = doc_co_occurrences[top, :][:, top].todense()
            occurrences = doc_occurrences[top]
            values = np.log(
                (co_occurrences * documents_number + co_occurrences_smooth)
                / (occurrences[:, np.newaxis] * occurrences[np.newaxis, :] + co_occurrences_smooth)
            )
            diag = np.diag_indices(len(values))
            values.cumsum(axis=0).cumsum(axis=1)[diag] - values[diag].cumsum()
            pmi += np.array(
               values.cumsum(axis=0).cumsum(axis=1)[diag] - values[diag].cumsum()
            ).ravel()

            values[values < 0.] = 0.
            ppmi += np.array(
               values.cumsum(axis=0).cumsum(axis=1)[diag] - values[diag].cumsum()
            ).ravel()

        sizes = np.arange(2, max_top_size + 1)
        pmi[1:] /= (T * sizes * (sizes - 1))
        ppmi[1:] /= (T * sizes * (sizes - 1))
        indices = np.array(top_sizes) - 1

        return pmi[indices], ppmi[indices]

    return func
```

```python
%%time

occurences, co_occurences = calc_doc_occurrences(dataset, MAIN_MODALITY)

calc_pmi = create_pmi_top_function(
    occurences, co_occurences,
    dataset.get_dataset().shape[0], [20],
    co_occurrences_smooth=1e-2,
)
```

    CPU times: user 9.41 s, sys: 1.27 s, total: 10.7 s
    Wall time: 10.6 s

```python
class TopTokenCoherence(BaseTopicNetScore):
    def __init__(self, func):
        super().__init__()

        self.calc_pmi = func

    def call(self, model: TopicModel):
        values = self.calc_pmi(model.get_phi_dense()[0].T)

        return values[1]
```

### Model training

```python
thetaless_reg = ThetalessRegularizer(
    name='thetaless',
    tau=1,
    dataset=dataset,
    modality=MAIN_MODALITY,
)

sparse_reg = artm.regularizers.SmoothSparsePhiRegularizer(
    name='sparse_reg',
    tau=-0.5,
)
```

```python
def create_model(dataset, modality: str):
    model = artm.ARTM(num_topics=30, num_document_passes=1, class_ids={modality: 1.0})
    model.initialize(dataset.get_dictionary())

    model.scores.add(
        artm.scores.PerplexityScore(name='perplexity', class_ids=[modality])
    )
    model.scores.add(
        artm.scores.SparsityPhiScore(name='sparsity', class_id=modality)
    )
    model.scores.add(
        artm.scores.TopicKernelScore(name='kernel', class_id=modality)
    )

    return model
```

```python
NUM_FIT_ITERATIONS = 50
```

```python
plsa_model = TopicModel(
    artm_model=create_model(dataset, MAIN_MODALITY),
    model_id='plsa_model',
    custom_scores={
        "coherence_ppmi": TopTokenCoherence(calc_pmi)
    }
)

%time plsa_model._fit(dataset.get_batch_vectorizer(), NUM_FIT_ITERATIONS)
```

    CPU times: user 4min 7s, sys: 9.59 s, total: 4min 16s
    Wall time: 1min 30s

```python
thetaless_model = TopicModel(
    artm_model=create_model(dataset, MAIN_MODALITY),
    model_id='thetaless_model',
    custom_regularizers={
        thetaless_reg.name: thetaless_reg
    },
    custom_scores={
        "coherence_ppmi": TopTokenCoherence(calc_pmi)
    }
)

%time thetaless_model._fit(dataset.get_batch_vectorizer(), NUM_FIT_ITERATIONS)
```

    CPU times: user 6min 4s, sys: 37.7 s, total: 6min 42s
    Wall time: 3min 18s

```python
artm_model = create_model(dataset, MAIN_MODALITY)
artm_model.regularizers.add(sparse_reg)

sparse_model = TopicModel(
    artm_model=artm_model,
    model_id='sparse_model',
    custom_scores={"coherence_ppmi": TopTokenCoherence(calc_pmi)}
)

%time sparse_model._fit(dataset.get_batch_vectorizer(), NUM_FIT_ITERATIONS)
```

    CPU times: user 4min 21s, sys: 27.5 s, total: 4min 48s
    Wall time: 1min 45s

```python
artm_model = create_model(dataset, MAIN_MODALITY)
artm_model.regularizers.add(sparse_reg)

sparse_thetaless_model = TopicModel(
    artm_model=artm_model,
    model_id='sparse_thetaless_model',
    custom_regularizers={
        thetaless_reg.name: thetaless_reg
    },
    custom_scores={
        "coherence_ppmi": TopTokenCoherence(calc_pmi)
    }
)

%time sparse_thetaless_model._fit(dataset.get_batch_vectorizer(), NUM_FIT_ITERATIONS)
```

    CPU times: user 7min 1s, sys: 1min 20s, total: 8min 21s
    Wall time: 3min 49s

## Quality Visualizing

```python
def plot_metric(view_name, value_name, title_name):
    plt.figure(figsize=(14,8))

    plt.subplot(2,2,1)
    values = np.array(getattr(thetaless_model.score_tracker[view_name], value_name))
    med = np.median(values)
    values[values > 5 * med] = np.nan
    last_thetaless = values[-1]

    plt.title(f'{title_name} with Thetaless regularizer')
    plt.plot(values[1:])

    plt.subplot(2,2,2)
    values = np.array(getattr(plsa_model.score_tracker[view_name], value_name))
    med = np.median(values)
    values[values > 5 * med] = np.nan
    last_plsa = values[-1]

    plt.title(f'{title_name} without any regularization')
    plt.plot(values[1:])

    plt.subplot(2,2,3)
    values = np.array(getattr(sparse_model.score_tracker[view_name], value_name))
    med = np.median(values)
    values[values > 5 * med] = np.nan
    last_sparse = values[-1]

    plt.title(f'{title_name} with sparsity regularizer')
    plt.plot(values[1:])

    plt.subplot(2,2,4)
    values = np.array(getattr(sparse_thetaless_model.score_tracker[view_name], value_name))
    med = np.median(values)
    values[values > 5 * med] = np.nan
    last_sparse_thetaless = values[-1]

    plt.title(f'{title_name} with both regularizers')
    plt.plot(values[1:])

    print(
        f'Last values:\n\t'
        f'thetaless – {last_thetaless}\n\t'
        f'plsa – {last_plsa}\n\t'
        f'sparse – {last_sparse}\n\t'
        f'sparse_thetaless – {last_sparse_thetaless}\n\t'
    )

    plt.show()
```

```python
plot_metric(view_name='perplexity', value_name='value', title_name='Perplexity')
```

```python
plot_metric(view_name='sparsity', value_name='value', title_name='Sparsity')
```

```python
plot_metric(view_name='kernel', value_name='average_contrast', title_name='Kernel Contrast')
```

```python
plot_metric(view_name='kernel', value_name='average_purity', title_name='Kernel purity')
```

```python
plot_metric(view_name='kernel', value_name='average_size', title_name='Kernel size')
```

```python
plot_metric(view_name='coherence_ppmi', value_name='value', title_name='Top-20 Coherence')
```

## Coherence evaluation on an external corpus (with Palmetto)

Unfortunately, the service isn't available right now

```python
def eval_top_words(name, model, num_top_tokens=10):
    ttv = TopTokensViewer(model, method='blei', num_top_tokens=num_top_tokens)
    output = ttv.view()
    S = "http://palmetto.aksw.org/palmetto-webapp/service/{}?words=".format("umass")
    topic_score = []

    for key, modalities in output.items():
        if 'topic' in key:
            words = list(output[key]['@default_class'].keys())

            print(S + "%20".join(words))

            result = requests.get(S + "%20".join(words)).text
            topic_score += [float(result)]

    print(round(np.mean(topic_score), 2), round(np.median(topic_score), 2))
```

```python
# eval_top_words('thetaless', thetaless_model)
```

```python
# eval_top_words('plsa', plsa_model)
```

```python
# eval_top_words('sparse', sparse_model)
```

```python
# eval_top_words('sparse_thetaless', sparse_thetaless_model)
```

## Let's look at topics

```python
def print_top_words(name, model, num_top_tokens=10):
    ttv = TopTokensViewer(model, method='blei', num_top_tokens=num_top_tokens)
    output = ttv.view()

    for key, modalities in output.items():
        if 'topic' in key:
            print(', '.join(output[key][MAIN_MODALITY].keys()))
            print()
```

```python
print_top_words('thetaless', thetaless_model)
```

    space, earth, launch, project, mission, nasa, satellite, station, orbit, degree

    car, light, bike, ground, engine, front, ride, mile, road, wire

    game, play, team, win, player, hit, season, lose, fan, league

    system, mail, send, list, software, computer, user, address, email, package

    find, question, many, post, read, write, name, follow, book, must

    also, high, power, large, small, low, offer, control, full, add

    key, information, chip, public, message, encryption, technology, security, clipper, algorithm

    run, work, window, thank, problem, please, help, set, appreciate, driver

    use, need, number, line, order, buy, price, sell, require, check

    drive, card, disk, mac, board, hard, mb, monitor, ms, memory

    study, drug, effect, medical, disease, patient, doctor, food, ed, health

    one, get, think, good, seem, really, put, hear, probably, much

    x, output, function, entry, widget, sun, motif, define, return, include

    believe, claim, fact, true, exist, evidence, agree, matter, argument, religion

    kill, armenian, war, armenians, turkish, attack, history, dead, jews, russian

    first, two, second, end, three, lead, hold, save, close, head

    time, new, year, come, day, call, last, old, great, still

    report, university, 1993, research, national, news, april, center, page, school

    state, government, law, gun, mr, president, country, us, house, member

    make, point, even, case, mean, keep, long, actually, consider, every

    1, 2, 0, 3, 4, 5, 6, 25, 10, 7

    say, people, right, never, life, child, man, person, live, feel

    would, know, like, could, tell, give, something, anyone, someone, anything

    8, 5, e, 7, 6, b, 9, c, 4, h

    g, w, r, q, p, v, k, u, z, n

    go, see, well, take, want, look, thing, way, try, back

    file, program, image, version, color, display, format, code, available, server

    god, jesus, church, christian, israel, bible, love, christ, sin, word

    ax, max, g9v, b8f, q, a86, 7, 145, 1d9, n

    may, also, since, part, group, change, however, problem, etc, cause

```python
print_top_words('plsa', plsa_model)
```

    space, launch, earth, project, mission, nasa, satellite, orbit, system, station

    car, bike, turn, power, light, ground, engine, back, front, ride

    year, game, last, win, play, lose, team, hit, player, guy

    system, mail, software, computer, list, user, pc, support, mac, package

    question, post, read, find, many, book, write, different, answer, article

    also, high, work, small, offer, large, time, several, low, interested

    key, public, information, message, chip, encryption, system, security, clipper, private

    window, run, thank, anyone, work, problem, help, please, use, get

    use, number, call, line, need, sell, price, may, order, phone

    drive, card, disk, hard, board, system, mb, speed, fast, scsi

    effect, cause, drug, study, medical, disease, patient, increase, doctor, food

    one, think, get, seem, good, really, hear, come, though, talk

    x, server, output, include, file, application, entry, widget, sun, program

    believe, people, true, exist, argument, religion, fact, matter, agree, belief

    mr, kill, people, armenian, q, armenians, president, turkish, say, attack

    first, right, two, point, second, give, keep, yes, end, pass

    go, could, new, time, get, ask, great, come, old, call

    report, university, 1993, april, national, research, page, include, school, center

    state, government, gun, law, country, control, us, member, crime, weapon

    make, even, may, case, consider, fire, less, mean, long, force

    25, db, team, play, period, hockey, goal, la, 10, 1st

    say, reason, claim, life, even, never, child, person, man, people

    would, know, like, something, tell, someone, good, bad, remember, enough

    1, 2, 0, 3, 4, 5, 6, 7, 8, 9

    w, g, r, b, c, k, v, p, u, e

    well, see, want, thing, look, try, go, make, take, let

    file, image, program, version, available, color, format, display, ftp, copy

    god, jesus, israel, word, jews, christ, bible, man, love, sin

    ax, max, q, g, r, b8f, g9v, 7, p, a86

    since, group, problem, etc, may, find, also, result, use, form

```python
print_top_words('sparse', sparse_model)
```

    space, center, science, research, launch, design, project, cost, material, mission

    get, good, back, turn, buy, need, stuff, little, bike, anybody

    year, game, last, team, play, win, player, lose, guy, next

    system, computer, software, mail, machine, user, mac, base, memory, monitor

    question, post, read, write, answer, must, article, different, message, bit

    number, also, high, small, large, several, open, case, use, type

    key, information, chip, use, group, provide, may, encryption, security, clipper

    use, work, problem, run, please, anyone, help, thank, need, set

    use, new, call, price, sell, offer, check, include, interested, box

    drive, card, hard, disk, power, speed, board, sound, driver, mb

    effect, condition, hit, soon, disease, patient, compare, mile, low, year

    think, one, seem, much, hear, though, idea, see, really, put

    x, list, send, line, copy, include, name, source, output, code

    believe, people, true, church, christian, argument, fact, israel, jews, agree

    kill, people, mr, president, armenian, child, armenians, q, attack, house

    first, second, two, name, body, save, three, third, face, mark

    go, take, come, could, time, start, leave, week, around, ago

    book, report, university, 1993, national, news, april, drug, page, turkish

    state, right, law, gun, government, people, issue, control, country, support

    point, long, yes, feel, make, less, may, never, mean, time

    10, 15, 12, 20, db, 11, 13, period, series, 30

    say, even, one, reason, right, claim, make, evidence, mind, perhaps

    would, know, like, tell, something, someone, car, really, wrong, remember

    1, 2, 0, 3, 4, 5, 6, 7, 8, 25

    g, w, r, b, p, c, q, n, v, e

    want, look, well, make, see, thing, try, way, let, like

    file, window, image, program, version, color, application, available, display, format

    god, man, word, jesus, life, world, christ, death, war, love

    ax, max, q, b8f, g9v, a86, 7, 145, 1d9, r

    find, since, part, many, cause, result, also, group, people, make

```python
print_top_words('sparse_thetaless', sparse_thetaless_model)
```

    space, earth, science, design, center, launch, field, project, material, mission

    get, good, back, power, buy, around, big, turn, stuff, light

    year, game, last, play, team, win, player, lose, guy, season

    system, mail, software, support, computer, machine, user, pc, standard, contact

    question, post, read, write, follow, must, ask, different, bit, note

    also, number, high, source, large, several, small, add, build, require

    key, information, chip, provide, phone, use, service, order, encryption, technology

    use, work, problem, need, run, please, help, thank, anyone, window

    new, call, include, price, etc, sell, offer, check, pay, cost

    drive, card, disk, driver, hard, speed, board, mb, fast, ms

    effect, condition, medical, soon, disease, compare, patient, mile, average, ed

    one, think, much, seem, really, still, put, hear, sure, probably

    x, send, list, line, copy, 25, output, return, section, request

    believe, people, true, fact, church, agree, argument, matter, israel, jews

    kill, mr, president, armenian, fire, child, armenians, attack, house, press

    first, two, second, lead, three, great, body, save, hit, john

    go, time, take, come, day, start, old, leave, next, week

    book, report, university, 1993, study, news, national, april, drug, page

    state, government, law, issue, gun, control, public, country, war, people

    may, point, case, mean, long, show, keep, course, actually, enough

    0, 10, 20, 15, 12, 14, 11, 16, 13, db

    say, even, right, people, never, reason, every, claim, person, live

    would, know, like, could, tell, give, something, someone, car, anything

    1, 2, 3, 4, 5, 6, 8, 7, 9, 27

    g, w, r, q, p, c, e, b, n, v

    make, see, well, want, look, thing, way, try, ca, let

    file, program, image, version, color, application, display, available, code, format

    god, man, word, life, world, jesus, christian, bible, love, christ

    ax, max, b8f, g9v, a86, q, 145, 1d9, 7, bhj

    find, many, since, part, group, however, cause, result, allow, free

## Last coherence values (using different number of top tokens)

The plots above show how the coherence (PPMI) of top 20 tokens evolves. In addition, we can look at the last PMI and PPMI values using different sizes of top-token list (5, 10, 20, 50, 100 words)

```python
occurences, co_occurences = calc_doc_occurrences(dataset, MAIN_MODALITY)

calc_pmi = create_pmi_top_function(
    occurences, co_occurences,
    dataset.get_dataset().shape[0], [5, 10, 20, 50, 100],
    co_occurrences_smooth=1e-2
)
```

```python
calc_pmi(plsa_model.get_phi_dense()[0].T)
```

    (array([1.21499297, 1.41841121, 1.4773908 , 1.64972323, 1.70756978]),
     array([1.31236076, 1.47163629, 1.54150819, 1.69634847, 1.80704321]))

```python
calc_pmi(thetaless_model.get_phi_dense()[0].T)
```

    (array([1.57331496, 1.62475971, 1.65381981, 1.86711067, 1.87356811]),
     array([1.57331496, 1.62499837, 1.69300155, 1.94859555, 2.09445053]))

```python
calc_pmi(sparse_model.get_phi_dense()[0].T)
```

    (array([1.35890928, 1.35700836, 1.48383579, 1.59996205, 1.60108423]),
     array([1.35890928, 1.37848896, 1.49985511, 1.65918859, 1.78920465]))

```python
calc_pmi(sparse_thetaless_model.get_phi_dense()[0].T)
```

    (array([1.42275237, 1.48636823, 1.59154109, 1.75142554, 1.55162446]),
     array([1.42275237, 1.48636823, 1.60343522, 1.83606053, 1.9786894 ]))

```python
# TODO: Also, let's take a look at train + test together, instead of relying on train only:
```

## Conclusion

We observed several interesting things.

1) the regularizer improves coherence and sparsity without explicitly optimizing for it

2) The regularizer moves common but uninformative words away from "informative" topics. Compare the 3rd topic of PLSA:

`year, game, last, win, play, lose, team, hit, player, guy`

with the corresponding topic of Thetaless:

`game, play, team, win, player, hit, season, lose, fan, league`

If we look at 20th topic, we will see the reverse trend.

The topic of PLSA is uninformative with a few meaningful words thrown in:

`make, even, may, case, consider, fire, less, mean, long, force`

The corresponding topic of Thetaless consist of common parlance almost entirely, without any signal lost in noise.

`make, point, even, case, mean, keep, long, actually, consider, every`

It should be noted that we had not specified the usual distincton between specific and background topics, the separation here is purely emergent.


# Gravity and Magnetic 3D Inversion

This application provides an interface to the open-source [SimPEG](https://simpeg.xyz/) package for the inversion of gravity and magnetic data (fields and tensor components). The application handles:

 - Gravity inversion for the recovery of residual density (g/cc).
 - Scalar magnetic inversion (MAG) for the recovery of susceptibility (SI).
 - Vector magnetic inversion (MVI) for the recovery of 3-component magnetization (effective susceptibility and angles).

All inversions are performed on a 3D octree mesh.

New user? Visit the [**Getting Started**](https://geoapps.readthedocs.io/en/latest/content/installation.html) page.

## Application
The following sections provide details on the different parameters controlling the application. Interactive widgets shown below are for demonstration purposes only.

```python
from geoapps.inversion.potential_fields.application import InversionApp

app = InversionApp(geoh5=r"../../../assets/FlinFlon.geoh5")
app()
```

    VBox(children=(VBox(children=(Label(value='Workspace', style=DescriptionStyle(description_width='initial')), H…

## Project Selection

Select and connect to an existing **geoh5** project file containing data

OR

Select a `*.ui.json` input file to re-load parameters from. See the [Input ui.json](#Input-ui.json) section for details.

```python
app.project_panel
```

    VBox(children=(Label(value='Workspace', style=DescriptionStyle(description_width='initial')), HBox(children=(F…

See the [Project Panel](base_application.ipynb#Project-Panel) page for more details.

## Survey Selection

### Object

List of objects available containing geophysical data to be inverted. Currently supports objects of type `Points`, `Curve`, `Surface` and `Grid2D`.

```python
app.objects
```

    Dropdown(description='Object:', index=18, options=(['', None], ['ISO/Iso_Iteration_7_model_5_00e-03', UUID('3f…

### Inversion Type

List of available inversion types.

- **gravity**: Invert bouguer corrected (mGal) or tensor (Etvos) data for the recovery of residual density (g/cc).
- **magnetic scalar**: Invert magnetic data for the recovery of scalar susceptibility (SI).
- **magnetic vector**: Invert magnetic data for the recovery of 3-component magnetization (effective susceptibility + angles).

```python
app.survey_type_panel
```

    HBox(children=(Dropdown(description='inversion Type:', options=('magnetic vector', 'magnetic scalar', 'gravity…

#### Inducing field parameters

For magnetic data only, set the inducing field parameters:

## Data Channel Options

Define the geophysical data to be inverted.

### Channel Options

Set parameters to invividual channels.

```python
app.data_channel_choices.data_channel_options['tmi']
```

    VBox(children=(Checkbox(value=True, description='Active', indent=False, style=DescriptionStyle(description_wid…

#### Active

Checked if the channel is to be used in the inversion

#### Channel:

Association between the "data channel" and one of the expected "system channel".

#### Uncertainty floor

Estimated data uncertainty floor (constant) value associated with the data channel.

#### Uncertainty channel

Optionally, use a data channel to assign point-based uncertainties. Uncertainty floor value is ignored if a channel of uncertainties is selected.

## Window Selection

Manual selection of an area of interest and data resolution.

```python
app.window_selection
```

    VBox(children=(VBox(children=(FloatText(value=50.0, description='Grid Resolution (m)', style=DescriptionStyle(…

See the [Map View Selection](view_selection.ipynb#Map-View-Selection) page for more details.

## Spatial information

### Topography

Set the air/ground interface of the inversion model.

```python
app.topography_group()
```

    VBox(children=(RadioButtons(description='Define by:', index=1, options=('None', 'Object', 'Relative to Sensor'…

#### Object

Set the topography based on an object stored in the target `geoh5` project. The Z value of the cells/vertices can be assigned based on a chosen `Data` field.

#### Relative to Sensor

Topography is defined by a fixed vertical offset from a selected object position (vertices or centroids), also referred to as a "draped height".

#### Constant

Topography is defined as a flat surface with constant elevation.

### Sensor Location

Defines the sensors position in 3D space. The vertical position can be referenced to:
- **sensor location**: Use the target [object](#Object) vertices or centroids z-locations
- **topo + radar**: Interpolate the topography to set the elevation. Typically used for gridded
    data with constant draped height or for airborne survey with inaccurate GPS
    elevation (see [radar channel](#Radar-channel-(optional))).

```python
app.sensor()
```

    VBox(children=(Checkbox(value=True, description='Set Z from topo + offsets', style=DescriptionStyle(descriptio…

#### Constant offsets (dx, dy, dz)

Sensor position shifted by a constant offset from the vertices of the selected
object. Typically used for towed systems where the GPS receiver is on the
aircraft.

- dx: East(+) or West(-) offset.
- dy: North(+) or South(-) offset
- dz: Up(+) or Down(-) offset

#### Radar channel (optional)

Channel defining the ground clearance (radar) of each data point.

## Inversion Options

List of parameters controlling the inversion.

```python
app.option_choices
```

    Dropdown(options=('starting model', 'mesh', 'reference model', 'regularization', 'upper-lower bounds', 'detren…

### Starting Model

Initial model used to begin the inversion.

```python
app._starting_model_group()
```

    VBox(children=(Label(value='Starting effective susceptibility', style=DescriptionStyle(description_width='init…

#### Model

Model object and values selected from any `Points`, `Curve`, `Surface`, `BlockModel` or `Octree` object.
Values are interpolated onto the inversion mesh using a nearest neighbor approach.

#### Constant

Constant background half-space value.

## Mesh

### Option 1: Object
Select an existing `Octree` mesh object from the target geoh5.

```python
app._mesh_octree.objects
```

    Dropdown(description='Object:', options=(['', None], ['ISO/Iso_Iteration_7_model_5_00e-03', UUID('3fde53cf-ea6…

### Option 2: Create

Set parameters used to build of an `Octree` mesh.

```python
app._mesh_octree._parameters
```

    VBox(children=(Label(value='Core cell size (u, v, z)'), FloatText(value=25.0, style=DescriptionStyle(descripti…

#### Core cell size (u, v, z)

Smallest (core) cell dimensions along

- u: East-West
- v: North-South
- z: Vertical

As a general rule of thumb, the cell size should be roughly equal or smaller to the data resolution.

####  # Cells below topography

Number of layers of cells at each octree level below the topography surface.
As topography generally extends further, it is common practice to use only
coarse cells once outside of the survey area.

####  # Cells below sensors

Number of layers of cells at each octree level below the observation points.

#### Maximum distance

Parameter controlling the Delaunay triangulation used for the refinement of
the mesh around topography and observation points. Large triangles generated
between points far apart are filtered out, reducing the number of small cells
in regions without data.

####  Horizontal padding

Additional padding distance (m) around the survey extent. Distances are rounded up to the next power of 2.

####  Vertical padding

Additional padding distance (m) added above and below the survey vertical relief. Distances rounded up to the next power of 2.

####  Minimum depth

Minimum thickness of the mesh added at depth below topography.

See the [Octree Mesh Creation](create_octree.ipynb#Mesh-Parameters) app for more details

## Regularization Panel

Parameters controlling the regularization function.

```python
app.inversion_options['regularization']
```

    HBox(children=(VBox(children=(Label(value='Scaling (alphas)'), FloatText(value=1.0, description='Reference Mod…

### Reference model

Reference model values used to constrain the inversion.

```python
app._reference_model_group()
```

    VBox(children=(Label(value='Reference effective susceptibility', style=DescriptionStyle(description_width='ini…

#### None

No reference model used. This is equivalent as setting the [Alphas](#Alpha-Parameters) `s` parameter to zero.

#### Model

Reference model selected from any `Points`, `Curve`, `Surface`, `BlockModel` or `Octree` object.
Values are interpolated onto the inversion mesh using a nearest neighbor approach.

#### Constant

Constant reference half-space value.

### Scaling (alpha) parameters

```python
app._alphas
```

    VBox(children=(Label(value='Scaling (alphas)'), FloatText(value=1.0, description='Reference Model (s)', style=…

Scaling between the components of the regularization function.

See the [SimPEG.API](https://docs.simpeg.xyz/content/api_core/api_Regularization.html#SimPEG.regularization.Simple) for technical details.

### $l_p$-norms

Approximated norms applied to the components of the regularization.

```python
app._norms
```

    VBox(children=(Label(value='Lp-norms'), FloatText(value=0.0, style=DescriptionStyle(description_width='initial…

See notes on [Sparse and Blocky Norms](https://giftoolscookbook.readthedocs.io/en/latest/content/fundamentals/Norms.html#sparse-and-blocky-norms) for technical details.

### Upper-Lower Bounds

Upper and lower bound constraints applied on physical property model.

```python
app.bound_panel
```

    HBox(children=(VBox(children=(Label(value='Lower Bounds'), VBox(children=(Label(value='', style=DescriptionSty…

Bounds can be defined as either:
- Model: Values defined on a cell-by-cell basis. The values are projected onto the inversion mesh using a nearest neighour interpolation.
- Constant: Constant value defining the bounds.
- None: The inversion uses [$-\infty$, $\infty$] as physical property bounds.

### Detrend (optional)

Remove a polynomial trend from the selected data channels.

```python
app.workspace

```

    <geoh5py.workspace.workspace.Workspace at 0x1da53c2e760>

```python
import os
app.write_trigger(None)
app.params.write_input_file(name="VectorInversion.ui.json", path='./')
app.detrend_type.value = "all"
```

```python
app._detrend_panel
```

    VBox(children=(Dropdown(description='Method', index=1, options=('', 'all', 'perimeter'), style=DescriptionStyl…

#### Order

Order [0, 1 or 2] of the polynomial defining the trend. The 0th order trend corresponds to removing the mean value.

#### Method

- all: Use all data points to compute the trend
- perimeter: Use only the edge points to compute the trend.

### Ignore Values

```python
app.ignore_values
```

    Text(value='', description="Value (i.e. '<0' for no negatives)", style=DescriptionStyle(description_width='ini…

Flag value ignored by the inversion by assigning $\infty$ uncertainties on the data points.

### Optimization

Parameters controlling various aspects of the projected Gauss-Newton inversion algorithm.

```python
app.optimization
```

    VBox(children=(IntText(value=25, description='Max beta Iterations', style=DescriptionStyle(description_width='…

#### Max beta Iterations

Maximum number of $\beta$-iterations allowed.
Note that when applying sparse norms, the inversion may require $>20$ iterations to converge.

#### Target misfit

Target data misfit where $\chi=1$ corresponds to $\phi_d=N$ (number of data). (See documentation on [Data Misfit and Uncertainties](https://giftoolscookbook.readthedocs.io/en/latest/content/fundamentals/Uncertainties.html#data-misfit-and-uncertainties) for mathematical details)

#### Starting trade-off ($\beta$)

**ratio**: Factor multiplying the initial $\beta$ defined by the ratio between the initial misfit and regularization:

\begin{equation}
\beta_0 = \gamma * \frac{\phi_d}{ \phi_m}
\end{equation}

**value**: Fixed $\beta$ value specified by the user.

#### Max CG Iterations

Maximum number of Conjugate Gradient (CG) iterations per Gauss-Newton solve.

#### CG Tolerance

Threshold on the minimum Conjugate Gradient (CG) step to end the Gauss-Newton solve.

#### Max CPUs

Maximum number of threads used for the parallelization. Defaults to half the system thread count.

#### Number of tiles

Number of data tiles used by the mesh decoupling algorithm. Nested sensitivities for each data blocks are interpolated to the global inversion mesh with a volumetric weighted averaging scheme.

## Output panel

Setup the inversion and launch the process.

### Output Name

```python
app.ga_group_name
```

    Text(value='VectorInversion', description='Save as:', style=DescriptionStyle(description_width='initial'))

Name given to the inversion group added to the Geoscience ANALYST project.

### Working directory

```python
app.export_directory
```

    FileChooser(path='C:\Users\dominiquef\Documents\GIT\mira\geoapps\docs\content\applications', filename='', titl…

Specify the working directory where the inversion occurs.

**It is highly recommended to chose a location on a solid-state drive (SSD) to fully take advantage of the `Dask` parallelization.**

### Write input

```python
app.write
```

    Button(description='Write input', icon='check', style=ButtonStyle())

Click to write the input parameters to a `*.json` file and a workspace (.geoh5) with all required entities.

### Run Inversion:

```python
app.trigger
```

    Button(button_style='success', description='Compute', icon='check', style=ButtonStyle(), tooltip='Run computat…

Launch the inversion routine. Results will be written directly to the target `geoh5` project.

## Input ui.json

This application relies on a [structured json](./VectorInversion.ui.json) file to store the parameters and run the program.
The `ui.json` can be used to re-load the parameters from a previous run by selecting the `ui.json` file from the [Project Selection](#Project-Selection) widget, instead of a `geoh5` file.

### Command line execution

The same input `ui.json` file can be used to run the program from command line:

```
activate geoapps
python -m geoapps.inversion.driver [Name].ui.json
```

This assumes that the geoapps have been installed and that the reference `*.ui.json` and `*.geoh5` project are accessible in the current directory.

### Geoscience ANALYST Pro v3.4

Geoscience ANALYST Pro users (v3.4) can execute this application directly from an active session with a drag & drop of a `*.ui.json` file to the `Viewport`.

```python
import matplotlib.pyplot as plt
app.figure.savefig("images/inversion_thumbnail.png", dpi=300, bbox_inches="tight")
```


[Sebastian Raschka](http://sebastianraschka.com)

[Link to](https://github.com/rasbt/pattern_classification) the GitHub repository [pattern_classification](https://github.com/rasbt/pattern_classification)

```python
%load_ext watermark
```

```python
%watermark -a 'Sebastian Raschka' -d -u -v -p scikit-learn,numpy,matplotlib
```

    Sebastian Raschka
    Last updated: 07/30/2015

    CPython 3.4.3
    IPython 3.2.1

    scikit-learn 0.16.1
    numpy 1.9.2
    matplotlib 1.4.3

I would be happy to hear your comments and suggestions.
Please feel free to drop me a note via
[twitter](https://twitter.com/rasbt), [email](mailto:bluewoodtree@gmail.com), or [google+](https://plus.google.com/+SebastianRaschka).

# About Feature Scaling: Z-Score Standardization and Min-Max Scaling

## Sections

- [About standardization](#About-standardization)
- [About Min-Max scaling / "normalization"](#About-Min-Max-scaling-normalization)
- [Standardization or Min-Max scaling?](#Standardization-or-Min-Max-scaling?)
- [Standardizing and normalizing - how it can be done using scikit-learn](#Standardizing-and-normalizing---how-it-can-be-done-using-scikit-learn)
- [Bottom-up approaches](#Bottom-up-approaches)
- [The effect of standardization on PCA in a pattern classification task](#The-effect-of-standardization-on-PCA-in-a-pattern-classification-task)

## About standardization

[[back to top](#Sections)]

The result of **standardization** (or **Z-score normalization**) is that the features will be rescaled so that they'll have the properties of a standard normal distribution with

$\mu = 0$ and $\sigma = 1$

where $\mu$ is the mean (average) and $\sigma$ is the standard deviation from the mean; standard scores (also called ***z*** scores) of the samples are calculated as follows:

\begin{equation} z = \frac{x - \mu}{\sigma}\end{equation}

Standardizing the features so that they are centered around 0 with a standard deviation of 1 is not only important if we are comparing measurements that have different units, but it is also a general requirement for many machine learning algorithms. Intuitively, we can think of gradient descent as a prominent example
(an optimization algorithm often used in logistic regression, SVMs, perceptrons, neural networks etc.); with features being on different scales, certain weights may update faster than others since the feature values $x_j$ play a role in the weight updates

$$\Delta w_j = - \eta \frac{\partial J}{\partial w_j} = \eta \sum_i (t^{(i)} - o^{(i)})x^{(i)}_{j},$$

so that

$$w_j := w_j + \Delta w_j,$$
where $\eta$ is the learning rate, $t$ the target class label, and $o$ the actual output.
Other intuitive examples include K-Nearest Neighbor algorithms and clustering algorithms that use, for example, Euclidean distance measures -- in fact, tree-based classifier are probably the only classifiers where feature scaling doesn't make a difference.

To quote from the [`scikit-learn`](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) documentation:

*"Standardization of a dataset is a common requirement for many machine learning estimators: they might behave badly if the individual feature do not more or less look like standard normally distributed data (e.g. Gaussian with 0 mean and unit variance)."*

## About Min-Max scaling

[[back to top](#Sections)]

An alternative approach to Z-score normalization (or standardization) is the so-called **Min-Max scaling** (often also simply called "normalization" - a common cause for ambiguities).
In this approach, the data is scaled to a fixed range - usually 0 to 1.
The cost of having this bounded range - in contrast to standardization - is that we will end up with smaller standard deviations, which can suppress the effect of outliers.

A Min-Max scaling is typically done via the following equation:

\begin{equation} X_{norm} = \frac{X - X_{min}}{X_{max}-X_{min}} \end{equation}

## Z-score standardization or Min-Max scaling?

[[back to top](#Sections)]

*"Standardization or Min-Max scaling?"* - There is no obvious answer to this question: it really depends on the application.

For example, in clustering analyses, standardization may be especially crucial in order to compare similarities between features based on certain distance measures. Another prominent example is the Principal Component Analysis, where we usually prefer standardization over Min-Max scaling, since we are interested in the components that maximize the variance (depending on the question and if the PCA computes the components via the correlation matrix instead of the covariance matrix; [but more about PCA in my previous article](http://sebastianraschka.com/Articles/2014_pca_step_by_step.html)).

However, this doesn't mean that Min-Max scaling is not useful at all! A popular application is image processing, where pixel intensities have to be normalized to fit within a certain range (i.e., 0 to 255 for the RGB color range). Also, typical neural network algorithm require data that on a 0-1 scale.

## Standardizing and normalizing - how it can be done using scikit-learn

[[back to top](#Sections)]

Of course, we could make use of NumPy's vectorization capabilities to calculate the z-scores for standardization and to normalize the data using the equations that were mentioned in the previous sections. However, there is an even more convenient approach using the preprocessing module from one of Python's open-source machine learning library [scikit-learn](http://scikit-learn.org ).

```python
%watermark -a 'Sebastian Raschka' -d -p scikit-learn,numpy,pandas,matplotlib -v
```

    Sebastian Raschka 12/07/2014

    CPython 3.4.1
    IPython 2.1.0

    scikit-learn 0.15.0b1
    numpy 1.8.1
    pandas 0.14.0
    matplotlib 1.3.1

[More information](http://nbviewer.ipython.org/github/rasbt/python_reference/blob/master/ipython_magic/watermark.ipynb) about the `watermark` magic command extension.

For the following examples and discussion, we will have a look at the free "Wine" Dataset that is deposited on the UCI machine learning repository
(http://archive.ics.uci.edu/ml/datasets/Wine).

**Reference:**
Forina, M. et al, PARVUS - An Extendible Package for Data
Exploration, Classification and Correlation. Institute of Pharmaceutical
and Food Analysis and Technologies, Via Brigata Salerno,
16147 Genoa, Italy.

Bache, K. & Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

The Wine dataset consists of 3 different classes where each row correspond to a particular wine sample.

The class labels (1, 2, 3) are listed in the first column, and the columns 2-14 correspond to 13 different attributes (features):

1) Alcohol
2) Malic acid
...

#### Loading the wine dataset

```python
import pandas as pd
import numpy as np

df = pd.io.parsers.read_csv(
    'https://raw.githubusercontent.com/rasbt/pattern_classification/master/data/wine_data.csv',
     header=None,
     usecols=[0,1,2]
    )

df.columns=['Class label', 'Alcohol', 'Malic acid']

df.head()
```

      Class label
      Alcohol
      Malic acid

      0
       1
       14.23
       1.71

      1
       1
       13.20
       1.78

      2
       1
       13.16
       2.36

      3
       1
       14.37
       1.95

      4
       1
       13.24
       2.59

As we can see in the table above, the features **Alcohol** (percent/volumne) and **Malic acid** (g/l) are measured on different scales, so that ***Feature Scaling*** is necessary important prior to any comparison or combination of these data.

#### Standardization and Min-Max scaling

```python
from sklearn import preprocessing

std_scale = preprocessing.StandardScaler().fit(df[['Alcohol', 'Malic acid']])
df_std = std_scale.transform(df[['Alcohol', 'Malic acid']])

minmax_scale = preprocessing.MinMaxScaler().fit(df[['Alcohol', 'Malic acid']])
df_minmax = minmax_scale.transform(df[['Alcohol', 'Malic acid']])
```

```python
print('Mean after standardization:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(df_std[:,0].mean(), df_std[:,1].mean()))
print('\nStandard deviation after standardization:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(df_std[:,0].std(), df_std[:,1].std()))
```

    Mean after standardization:
    Alcohol=0.00, Malic acid=0.00

    Standard deviation after standardization:
    Alcohol=1.00, Malic acid=1.00

```python
print('Min-value after min-max scaling:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(df_minmax[:,0].min(), df_minmax[:,1].min()))
print('\nMax-value after min-max scaling:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(df_minmax[:,0].max(), df_minmax[:,1].max()))
```

    Min-value after min-max scaling:
    Alcohol=0.00, Malic acid=0.00

    Max-value after min-max scaling:
    Alcohol=1.00, Malic acid=1.00

#### Plotting

```python
%matplotlib inline
```

```python
from matplotlib import pyplot as plt

def plot():
    plt.figure(figsize=(8,6))

    plt.scatter(df['Alcohol'], df['Malic acid'],
            color='green', label='input scale', alpha=0.5)

    plt.scatter(df_std[:,0], df_std[:,1], color='red',
            label='Standardized [$N  (\mu=0, \; \sigma=1)$]', alpha=0.3)

    plt.scatter(df_minmax[:,0], df_minmax[:,1],
            color='blue', label='min-max scaled [min=0, max=1]', alpha=0.3)

    plt.title('Alcohol and Malic Acid content of the wine dataset')
    plt.xlabel('Alcohol')
    plt.ylabel('Malic Acid')
    plt.legend(loc='upper left')
    plt.grid()

    plt.tight_layout()

plot()
plt.show()
```

The plot above includes the wine datapoints on all three different scales: the input scale where the alcohol content was measured in volume-percent (green), the standardized features (red), and the normalized features (blue).
In the following plot, we will zoom in into the three different axis-scales.

```python
fig, ax = plt.subplots(3, figsize=(6,14))

for a,d,l in zip(range(len(ax)),
               (df[['Alcohol', 'Malic acid']].values, df_std, df_minmax),
               ('Input scale',
                'Standardized [$N  (\mu=0, \; \sigma=1)$]',
                'min-max scaled [min=0, max=1]')
                ):
    for i,c in zip(range(1,4), ('red', 'blue', 'green')):
        ax[a].scatter(d[df['Class label'].values == i, 0],
                  d[df['Class label'].values == i, 1],
                  alpha=0.5,
                  color=c,
                  label='Class %s' %i
                  )
    ax[a].set_title(l)
    ax[a].set_xlabel('Alcohol')
    ax[a].set_ylabel('Malic Acid')
    ax[a].legend(loc='upper left')
    ax[a].grid()

plt.tight_layout()

plt.show()
```

## Bottom-up approaches

[[back to top](#Sections)]

Of course, we can also code the equations for standardization and 0-1 Min-Max scaling "manually". However, the scikit-learn methods are still useful if you are working with test and training data sets and want to scale them equally.

E.g.,

std_scale = preprocessing.StandardScaler().fit(X_train)
X_train = std_scale.transform(X_train)
X_test = std_scale.transform(X_test)

Below, we will perform the calculations using "pure" Python code, and an more convenient NumPy solution, which is especially useful if we attempt to transform a whole matrix.

Just to recall the equations that we are using:

Standardization: \begin{equation} z = \frac{x - \mu}{\sigma} \end{equation}

with mean:

\begin{equation}\mu = \frac{1}{N} \sum_{i=1}^N (x_i)\end{equation}

and standard deviation:

\begin{equation}\sigma = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \mu)^2}\end{equation}

Min-Max scaling: \begin{equation} X_{norm} = \frac{X - X_{min}}{X_{max}-X_{min}} \end{equation}

### Pure Python

```python
# Standardization

x = [1,4,5,6,6,2,3]
mean = sum(x)/len(x)
std_dev = (1/len(x) * sum([ (x_i - mean)**2 for x_i in x]))**0.5

z_scores = [(x_i - mean)/std_dev for x_i in x]

# Min-Max scaling

minmax = [(x_i - min(x)) / (max(x) - min(x)) for x_i in x]
```

### NumPy

```python
import numpy as np

# Standardization

x_np = np.asarray(x)
z_scores_np = (x_np - x_np.mean()) / x_np.std()

# Min-Max scaling

np_minmax = (x_np - x_np.min()) / (x_np.max() - x_np.min())
```

### Visualization

Just to make sure that our code works correctly, let us plot the results via matplotlib.

```python
%matplotlib inline
```

```python
from matplotlib import pyplot as plt

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(nrows=2, ncols=2, figsize=(10,5))

y_pos = [0 for i in range(len(x))]

ax1.scatter(z_scores, y_pos, color='g')
ax1.set_title('Python standardization', color='g')

ax2.scatter(minmax, y_pos, color='g')
ax2.set_title('Python Min-Max scaling', color='g')

ax3.scatter(z_scores_np, y_pos, color='b')
ax3.set_title('Python NumPy standardization', color='b')

ax4.scatter(np_minmax, y_pos, color='b')
ax4.set_title('Python NumPy Min-Max scaling', color='b')

plt.tight_layout()

for ax in (ax1, ax2, ax3, ax4):
    ax.get_yaxis().set_visible(False)
    ax.grid()

plt.show()
```

## The effect of standardization on PCA in a pattern classification task

[[back to top](#Sections)]

Earlier, I mentioned the Principal Component Analysis (PCA) as an example where standardization is crucial, since it is "analyzing" the variances of the different features.
Now, let us see how the standardization affects PCA and a following supervised classification on the whole wine dataset.

In the following section, we will go through the following steps:

- Reading in the dataset
- Dividing the dataset into a separate training and test dataset
- Standardization of the features
- Principal Component Analysis (PCA) to reduce the dimensionality
- Training a naive Bayes classifier
- Evaluating the classification accuracy with and without standardization

### Reading in the dataset

[[back to top](#Sections)]

```python
import pandas as pd

df = pd.io.parsers.read_csv(
    'https://raw.githubusercontent.com/rasbt/pattern_classification/master/data/wine_data.csv',
    header=None,
    )
```

### Dividing the dataset into a separate training and test dataset

[[back to top](#Sections)]

In this step, we will randomly divide the wine dataset into a training dataset and a test dataset where the training dataset will contain 70% of the samples and the test dataset will contain 30%, respectively.

```python
from sklearn.cross_validation import train_test_split

X_wine = df.values[:,1:]
y_wine = df.values[:,0]

X_train, X_test, y_train, y_test = train_test_split(X_wine, y_wine,
    test_size=0.30, random_state=12345)
```

### Feature Scaling - Standardization

[[back to top](#Sections)]

```python
from sklearn import preprocessing

std_scale = preprocessing.StandardScaler().fit(X_train)
X_train_std = std_scale.transform(X_train)
X_test_std = std_scale.transform(X_test)
```

### Dimensionality reduction via Principal Component Analysis (PCA)

[[back to top](#Sections)]

Now, we perform a PCA on the standardized and the non-standardized datasets to transform the dataset onto a 2-dimensional feature subspace.
In a real application, a procedure like cross-validation would be done in order to find out what choice of features would yield a optimal balance between "preserving information" and "overfitting" for different classifiers. However, we will omit this step since we don't want to train a perfect classifier here, but merely compare the effects of standardization.

```python
from sklearn.decomposition import PCA

# on non-standardized data
pca = PCA(n_components=2).fit(X_train)
X_train = pca.transform(X_train)
X_test = pca.transform(X_test)

# om standardized data
pca_std = PCA(n_components=2).fit(X_train_std)
X_train_std = pca_std.transform(X_train_std)
X_test_std = pca_std.transform(X_test_std)
```

Let us quickly visualize how our new feature subspace looks like (note that class labels are not considered in a PCA - in contrast to a Linear Discriminant Analysis - but I will add them in the plot for clarity).

```python
%matplotlib inline
```

```python
from matplotlib import pyplot as plt

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10,4))

for l,c,m in zip(range(1,4), ('blue', 'red', 'green'), ('^', 's', 'o')):
    ax1.scatter(X_train[y_train==l, 0], X_train[y_train==l, 1],
        color=c,
        label='class %s' %l,
        alpha=0.5,
        marker=m
        )

for l,c,m in zip(range(1,4), ('blue', 'red', 'green'), ('^', 's', 'o')):
    ax2.scatter(X_train_std[y_train==l, 0], X_train_std[y_train==l, 1],
        color=c,
        label='class %s' %l,
        alpha=0.5,
        marker=m
        )

ax1.set_title('Transformed NON-standardized training dataset after PCA')
ax2.set_title('Transformed standardized training dataset after PCA')

for ax in (ax1, ax2):

    ax.set_xlabel('1st principal component')
    ax.set_ylabel('2nd principal component')
    ax.legend(loc='upper right')
    ax.grid()
plt.tight_layout()

plt.show()
```

### Training a naive Bayes classifier

[[back to top](#Sections)]

We will use a naive Bayes classifier for the classification task. If you are not familiar with it, the term "naive" comes from the assumption that all features are "independent".
All in all, it is a simple but robust classifier based on Bayes' rule

Bayes' Rule:

\begin{equation} P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)} \end{equation}

where

- ω:  class label
- *P(ω|x)*: the posterior probability
- *p(x|ω)*: prior probability (or likelihood)

and the **decsion rule:**

Decide $ \omega_1 $ if $ P(\omega_1|x) > P(\omega_2|x) $ else decide $ \omega_2 $.

\begin{equation}
\Rightarrow \frac{p(x|\omega_1) * P(\omega_1)}{p(x)} > \frac{p(x|\omega_2) * P(\omega_2)}{p(x)}
\end{equation}

I don't want to get into more detail about Bayes' rule in this article, but if you are interested in a more detailed collection of examples, please have a look at the [Statistical Patter Classification](https://github.com/rasbt/pattern_classification#statistical-pattern-recognition-examples) in my pattern classification repository.

```python
from sklearn.naive_bayes import GaussianNB

# on non-standardized data
gnb = GaussianNB()
fit = gnb.fit(X_train, y_train)

# on standardized data
gnb_std = GaussianNB()
fit_std = gnb_std.fit(X_train_std, y_train)
```

### Evaluating the classification accuracy with and without standardization

[[back to top](#Sections)]

```python
from sklearn import metrics

pred_train = gnb.predict(X_train)

print('\nPrediction accuracy for the training dataset')
print('{:.2%}'.format(metrics.accuracy_score(y_train, pred_train)))

pred_test = gnb.predict(X_test)

print('\nPrediction accuracy for the test dataset')
print('{:.2%}\n'.format(metrics.accuracy_score(y_test, pred_test)))
```

    Prediction accuracy for the training dataset
    81.45%

    Prediction accuracy for the test dataset
    64.81%

```python
pred_train_std = gnb_std.predict(X_train_std)

print('\nPrediction accuracy for the training dataset')
print('{:.2%}'.format(metrics.accuracy_score(y_train, pred_train_std)))

pred_test_std = gnb_std.predict(X_test_std)

print('\nPrediction accuracy for the test dataset')
print('{:.2%}\n'.format(metrics.accuracy_score(y_test, pred_test_std)))
```

    Prediction accuracy for the training dataset
    96.77%

    Prediction accuracy for the test dataset
    98.15%

As we can see, the standardization prior to the PCA definitely led to an decrease in the empirical error rate on classifying samples from test dataset.