# Data Space Report

## Pittsburgh Bridges Data Set

    Andy Warhol Bridge - Pittsburgh.

Report created by Student Francesco Maria Chiarlo s253666, for A.A 2019/2020.

**Abstract**:The aim of this report is to evaluate the effectiveness of distinct, different statistical learning approaches, in particular focusing on their characteristics as well as on their advantages and backwards when applied onto a relatively small dataset as the one employed within this report, that is Pittsburgh Bridgesdataset.

**Key words**:Statistical Learning, Machine Learning, Bridge Design.

## TOC:
* [Imports Section](#imports-section)
* [Dataset's Attributes Description](#attributes-description)
* [Data Preparation and Investigation](#data-preparation)
* [Learning Models](#learning-models)
* [Improvements and Conclusions](#improvements-and-conclusions)
* [References](#references)

### Imports Section

```python
# =========================================================================== #
# STANDARD IMPORTS
# =========================================================================== #
print(__doc__)

from pprint import pprint

import warnings
warnings.filterwarnings('ignore')

import copy
import os
import sys
import time

import pandas as pd
import numpy as np

%matplotlib inline
# Matplotlib pyplot provides plotting API
import matplotlib as mpl
from matplotlib import pyplot as plt
import chart_studio.plotly.plotly as py
import seaborn as sns; sns.set()
```

```python
# =========================================================================== #
# UTILS IMPORTS (Done by myself)
# =========================================================================== #
from utils.display_utils import display_heatmap
from utils.display_utils import show_frequency_distribution_predictors
from utils.display_utils import show_frequency_distribution_predictor
from utils.display_utils import build_boxplot
from utils.display_utils import show_categorical_predictor_values
from utils.display_utils import  show_cum_variance_vs_components
from utils.display_utils import show_histograms_from_heatmap_corr_matrix

from utils.preprocessing_utils import preprocess_categorical_variables
from utils.preprocessing_utils import  preprocessing_data_rescaling

from utils.training_utils import sgd_classifier_grid_search
from utils.training_utils import naive_bayes_classifier_grid_search
from utils.training_utils import svm_linear_classifier_grid_search
from utils.training_utils import decision_tree_classifier_grid_search
from utils.training_utils import random_forest_classifier_grid_search
from utils.training_utils import plot_roc_crossval
```

```python
# =========================================================================== #
# sklearn IMPORT
# =========================================================================== #
from sklearn.decomposition import PCA
```

### Dataset's Attributes Description

The analyses that I aim at accomplishing while using as means the methods or approaches provided by both Statistical Learning and Machine Learning fields, concern the dataset Pittsburgh Bridges, and what follows is a overview and brief description of the main characteristics, as well as, basic information about this precise dataset.

The Pittsburgh Bridges dataset is a dataset available from the web site called mainly *"UCI Machine Learing Repository"*, which is one of the well known web site that let a large amount of different datasets, from different domains or fields, to be used for machine-learning research and which have been cited in peer-reviewed academic journals.

In particular, the dataset I'm going to treat and analyze, which is Pittsburgh Bridges dataset, has been made freely available from the Western Pennsylvania Regional Data Center (WPRDC), which is a project led by the University Center of Social and Urban Research (UCSUR) at the University of Pittsburgh ("University") in collaboration with City of Pittsburgh and The County of Allegheny in Pennsylvania. The WPRDC and the WPRDC Project is supported by a grant from the Richard King Mellon Foundation.

In order to be more precise, from the official and dedicated web page, within UCI Machine Learning cite, Pittsburgh Bridges dataset is a dataset that has been created after the works of some co-authors which are:
- Yoram Reich & Steven J. Fenves from Department of Civil Engineering and Engineering Design Research Center Carnegie Mellon University Pittsburgh, PA 15213

The Pittsburgh Bridges dataset is made of up to 108 distinct observations and each of that data sample is made of 12 attributes or features where some of them are considered to be continuous properties and other to be categorical or nominal properties. Those variables are the following:

- **RIVER**: which is a nominal type variable that can assume the subsequent possible discrete values which are: A, M, O. Where A stands for Allegheny river, while M stands for Monongahela river and lastly O stands for Ohio river.
- **LOCATION**: which represents a nominal type variable too, and assume a positive integer value from 1 up to 52 used as categorical attribute.
- **ERECTED**: which might be either a numerical or categorical variable, depending on the fact that we want to aggregate a bunch of value under a categorical quantity. What this means is that, basically such attribute is made of date starting from 1818 up to 1986, but we may imagine to aggregate somehow these data within a given category among those suggested, that are CRAFTS, EMERGENING, MATURE, MODERN.
- **PURPOSE**: which is a categorical attribute and represents the reason why a particular bridge has been built, which means that this attribute represents what kind of vehicle can cross the bridge or if the bridge has been made just for people. For this reasons the allowd values for this attributes are the following: WALK, AQUEDUCT, RR, HIGHWAY. Three out of four are self explained values, while RR value that might be tricky at first glance, it just stands for railroad.
- **LENGTH**: which represents the bridge's length, is a numerical attribute if we just look at the real number values that go from 804 up to 4558, but we can again decide to handle or arrange such values so that they can be grouped into range of values mapped into SHORT, MEDIUM, LONG so that we can refer to a bridge's length by means of these new categorical values.
- **LANES**: which is a categorical variable which is represented by numerical values, that are 1, 2, 4, 6 which indicate the number of distinct lanes that a bridge in Pittsburgh city may have. The larger the value the wider the bridge.
- **CLEAR-G**: specifies  whether  a  vertical  navigation clearance requirement was enforced in the design or not.
- **T-OR-D**: which is a nominal attribute, in other words, a categorical attribute that can assume THROUGH, DECK values. In order to be more precise, this samples attribute deals with structural elements of a bridge. In fact, a deck is the surface of a bridge and this structural element, of bridge's superstructure, may be constructed of concrete, steel, open grating, or wood. On the other hand, a through arch bridge, also known as a half-through arch bridge or a through-type arch bridge, is a bridge that is made from materials such as steel or reinforced concrete, in which the base of an arch structure is below the deck but the top rises above it.
- **MATERIAL**: which is a categorical or nominal variable and is used to describe the bridge telling which is the main or core material used to build it.
  This attribute can assume one of the possible, following values which are: WOOD, IRON, STEEL. Furthermore, we expect to see somehow a bit of correlation between the values assumed by the pairs represented by T-OR-D and MATERIAL columns, when looking just to them.
- **SPAN**: which is a categorical or nominal value and has been recorded by means of three possible values for each sample, that are SHORT, MEDIUM, LONG. This attribute, within the field of Structural Engineering, is the distance between two intermediate supports for a structure, e.g. a beam or a bridge. A span can be closed by a solid beam or by a rope. The first kind is used for bridges, the second one for power lines, overhead telecommunication lines, some type of antennas or for aerial tramways.
- **REL-L**: which is a categorical or nominal variable and stands for relative length of the main span of the bridge to the total crossing length, it can assume three possible values that are S, S-F, F.
- Lastly, **TYPE** which indicates as a categorical or nominal attributes what type of bridge each record represents, among the possible 6 distinct classes or types of bridges that are: WOOD, SUSPEN, SIMPLE-T, ARCH, CANTILEV, CONT-T.

### Data Preparation and Investigation

The aim of this chapter is to get in the data, that are available within Pittsburgh Bridge Dataset, in order to investigate a bit more in to detail and generally speaking deeper the main or high level statistics quantities, such as mean, median, standard deviation of each attribute, as well as displaying somehow data distribution for each attribute by means of histogram plots. This phase allows or enables us to decide which should be the best feature to be selected as the target variable, in other word the attribute that will represent the dependent variable with respect to the remaining attributes that instead will play the role of predictors and independent variables, as well.

In order to investigate and explore our data we make usage of *Pandas library*. We recall mainly that, in computer programming, Pandas is a software library written for the Python programming language* for *data manipulation and analysis*. In particular, it offers data structures and operations for manipulating numerical tables and time series. It is free software and a interesting and funny things about such tool is that the name is derived from the term "panel data", an econometrics term for data sets that include observations over multiple time periods for the same individuals.
We also note that as the analysis proceeds we will introduce other computer programming as well as programming libraries that allow or enable us to fulfill our goals.

Initially, once I have downloaded from the provided web page the dataset with the data samples about Pittsburgh Bridge we load the data by means of functions available using python library's pandas. We notice that the overall set of data points is large up to 108 records or rows, which are sorted by Erected attributes, so this means that are sorted in decreasing order from the oldest bridge which has been built in 1818 up to the most modern bridge that has been erected in 1986. Then we display the first 5 rows to get an overview and have a first idea about what is inside the overall dataset, and the result we obtain by means of head() function applied onto the fetched dataset is equals to what follows:

```python
# =========================================================================== #
# READ INPUT DATASET
# =========================================================================== #

dataset_path = 'C:\\Users\\Francesco\Documents\\datasets\\pittsburgh_dataset'
dataset_name = 'bridges.data.csv'

# column_names = ['IDENTIF', 'RIVER', 'LOCATION', 'ERECTED', 'PURPOSE', 'LENGTH', 'LANES', 'CLEAR-G', 'T-OR-D', 'MATERIAL', 'SPAN', 'REL-L', 'TYPE']
column_names = ['RIVER', 'LOCATION', 'ERECTED', 'PURPOSE', 'LENGTH', 'LANES', 'CLEAR-G', 'T-OR-D', 'MATERIAL', 'SPAN', 'REL-L', 'TYPE']
dataset = pd.read_csv(os.path.join(dataset_path, dataset_name), names=column_names, index_col=0)
```

```python
# SHOW SOME STANDARD DATASET INFOS
# --------------------------------------------------------------------------- #
print('Dataset shape: {}'.format(dataset.shape))
print(dataset.info())
```

```python
# SHOWING FIRSTS N-ROWS AS THEY ARE STORED WITHIN DATASET
# --------------------------------------------------------------------------- #
dataset.head(5)
```

What we can notice from just the table above is that there are some attributes that are characterized by a special character that is '?' which stands for a missing value, so by chance there was not possibility to get the value for this attribute, such as for LENGTH and SPAN attributes. Analyzing in more details the dataset we discover that there are up to 6 different attributes, in the majority attributes with categorical or nominal nature such as CLEAR-G, T-OR-D, MATERIAL, SPAN, REL-L, and TYPE that contain at list one row characterized by the fact that one of its attributes is set to assuming '?' value that stands, as we already know for a missing value.

Here, we can follow different strategies that depends onto the level of complexity as well as accuracy we want to obtain or achieve for models we are going to fit to the data after having correctly pre-processed them, speaking about what we could do with missing values. In fact one can follow the simplest way and can decide to simply discard those rows that contain at least one attribute with a missing value represented by the '?' symbol. Otherwise one may alos decide to follow a different strategy that aims at keeping also those rows that have some missing values by means of some kind of technique that allows to establish a potential substituting value for the missing one.

So, in this setting, that is our analyses, we start by just leaving out those rows that at least contain one attribute that has a missing value, this choice leads us to reduce the size of our dataset from 108 records to 70 remaining samples, with a drop of 38 data examples, which may affect the final results, since we left out more or less the 46\% of the data because of missing values.

```python
# INVESTIGATING DATASET IN ORDER TO DETECT NULL VALUES
# --------------------------------------------------------------------------- #
print('Before preprocessing dataset and handling null values')
result = dataset.isnull().values.any()
print('There are any null values ? Response: {}'.format(result))

result = dataset.isnull().sum()
print('Number of null values for each predictor:\n{}'.format(result))
```

```python
# DISCOVERING VALUES WITHIN EACH PREDICTOR DOMAIN
# --------------------------------------------------------------------------- #
columns_2_avoid = ['ERECTED', 'LENGTH', 'LOCATION', 'LANES']
# columns_2_avoid = None
list_columns_2_fix = show_categorical_predictor_values(dataset, columns_2_avoid)
```

```python
# FIXING, UPDATING NULL VALUES CODED AS '?' SYMBOL
# WITHIN EACH CATEGORICAL VARIABLE, IF DETECTED ANY
# --------------------------------------------------------------------------- #
print('"Before" removing \'?\' rows, Dataset dim:', dataset.shape)
for _, predictor in enumerate(list_columns_2_fix):
    dataset = dataset[dataset[predictor] != '?']
print('"After" removing \'?\' rows, Dataset dim: ', dataset.shape)
print('-' * 50)

_ = show_categorical_predictor_values(dataset, columns_2_avoid)
```

```python
# INTERMEDIATE RESULT FOUND
# --------------------------------------------------------------------------- #
preprocess_categorical_variables(dataset, columns_2_avoid)
print(dataset.info())
```

```python
dataset.head(5)
```

The next step is represented by the effort of mapping categorical variables into numerical variables, so that them are comparable with the already existing numerical or continuous variables, and also by mapping the categorical variables into numerical variables we allow or enable us to perform some kind of normalization or just transformation onto the entire dataset in order to let some machine learning algorithm to work better or to take advantage of normalized data within our pre-processed dataset. Furthermore, by transforming first the categorical attributes into a continuous version we are also able to calculate the \textit{heatmap}, which is a very useful way of representing a correlation matrix calculated on the whole dataset. Moreover we have displayed data distribution for each attribute by means of histogram representation to take some useful information about the number of occurrences for each possible value, in particular for those attributes that have a categorical nature.

```python
# MAP NUMERICAL VALUES TO INTEGER VALUES
# --------------------------------------------------------------------------- #
print('Before', dataset.shape)
columns_2_map = ['ERECTED', 'LANES']
for _, predictor in enumerate(columns_2_map):
    dataset = dataset[dataset[predictor] != '?']
    dataset[predictor] = np.array(list(map(lambda x: int(x), dataset[predictor].values)))
print('After', dataset.shape)
print(dataset.info())
print(dataset.head(5))
```

```python
# MAP NUMERICAL VALUES TO FLOAT VALUES
# --------------------------------------------------------------------------- #
# print('Before', dataset.shape)
columns_2_map = ['LOCATION', 'LANES', 'LENGTH']
for _, predictor in enumerate(columns_2_map):
    dataset = dataset[dataset[predictor] != '?']
    dataset[predictor] = np.array(list(map(lambda x: float(x), dataset[predictor].values)))
# print('After', dataset.shape)
# print(dataset.info())
# print(dataset.head(5))

# columns_2_avoid = None
# list_columns_2_fix = show_categorical_predictor_values(dataset, None)
```

```python
result = dataset.isnull().values.any()
# print('After handling null values\nThere are any null values ? Response: {}'.format(result))

result = dataset.isnull().sum()
# print('Number of null values for each predictor:\n{}'.format(result))

dataset.head(5)
```

```python
dataset.describe(include='all')
```

```python
# sns.pairplot(dataset, hue='T-OR-D', size=1.5)
```

```python
columns_2_avoid = ['ERECTED', 'LENGTH', 'LOCATION']
target_col = 'T-OR-D'
# show_frequency_distribution_predictors(dataset, columns_2_avoid)
show_frequency_distribution_predictor(dataset, predictor_name='RIVER', columns_2_avoid=columns_2_avoid)

build_boxplot(dataset, predictor_name='RIVER', columns_2_avoid=columns_2_avoid, target_col='T-OR-D')
```

```python
# show_frequency_distribution_predictors(dataset, columns_2_avoid)
show_frequency_distribution_predictor(dataset, predictor_name='T-OR-D', columns_2_avoid=columns_2_avoid)
```

```python
# show_frequency_distribution_predictors(dataset, columns_2_avoid)
show_frequency_distribution_predictor(dataset, predictor_name='CLEAR-G', columns_2_avoid=columns_2_avoid)

build_boxplot(dataset, predictor_name='CLEAR-G', columns_2_avoid=columns_2_avoid, target_col='T-OR-D')
```

```python
# show_frequency_distribution_predictors(dataset, columns_2_avoid)
show_frequency_distribution_predictor(dataset, predictor_name='SPAN', columns_2_avoid=columns_2_avoid)

build_boxplot(dataset, predictor_name='SPAN', columns_2_avoid=columns_2_avoid, target_col='T-OR-D')
```

```python
# show_frequency_distribution_predictors(dataset, columns_2_avoid)
show_frequency_distribution_predictor(dataset, predictor_name='MATERIAL', columns_2_avoid=columns_2_avoid)

build_boxplot(dataset, predictor_name='MATERIAL', columns_2_avoid=columns_2_avoid, target_col='T-OR-D')
```

```python
# show_frequency_distribution_predictors(dataset, columns_2_avoid)
show_frequency_distribution_predictor(dataset, predictor_name='REL-L', columns_2_avoid=columns_2_avoid)
```

```python
# show_frequency_distribution_predictors(dataset, columns_2_avoid)
show_frequency_distribution_predictor(dataset, predictor_name='TYPE', columns_2_avoid=columns_2_avoid)

build_boxplot(dataset, predictor_name='TYPE', columns_2_avoid=columns_2_avoid, target_col='T-OR-D')
```

```python
corr_result = dataset.corr()
display_heatmap(corr_result)
```

```python
print(type(corr_result))
corr_result.head(5)
```

```python
# show_histograms_from_heatmap_corr_matrix(corr_result, row_names=dataset.columns)
```

```python
columns = dataset.columns
target_col = 'T-OR-D'

y = np.array(list(map(lambda x: 0 if x == 1 else 1, dataset[target_col].values)))
print(dataset['T-OR-D'].value_counts())
X = dataset.loc[:, dataset.columns != target_col]
```

```python
# Standardizing the features
# --------------------------------------------------------------------------- #
scaler_methods = ['minmax', 'standard', 'norm']
scaler_method = 'standard'
rescaledX = preprocessing_data_rescaling(scaler_method, X)
```

### Pricipal Component Analysis

After having investigate the data points inside the dataset, I move one to another section of my report where I decide to explore examples that made up the entire dataset using a particular technique in the field of statistical analysis that corresponds, precisely, to so called Principal Component Analysis. In fact, the major objective of this section is understand whether it is possible to transform, by means of some kind of linear transformation given by a mathematical calculation, the original data examples into reprojected representation that allows me to retrieve most useful information to be later exploited at training time. So, lets dive a bit whitin what is and which are main concepts, pros and cons about Principal Component Analysis.

Firstly, we know that **Principal Component Analysis**, more shortly PCA, is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called *principal components*. This transformation is defined in such a way that:
- the first principal component has the largest possible variance (that is, accounts for as much of the variability in the data as possible),
- and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to the preceding components.

The resulting vectors, each being a linear combination of the variables and containing n observations, are an uncorrelated orthogonal basis set. PCA is sensitive to the relative scaling of the original variables.

PCA is mostly used as a tool in *exploratory data analysis* and for making predictive models, for that reasons I used such a technique here, before going through the different learning technique for producing my models.

#### Several Different Implementation

From the theory and the filed of research in statistics, we know that out there, there are several different implementation and way of computing principal component analysis, and each adopted technique has different performance as well as numerical stability. The three major derivations are:
- PCA by means of an iterative based procedure of extraing pricipal components one after the other selecting each time the one that account for the most of variance along its own axis, within the remainig subspace to be derived.
- The second possible way of performing PCA is done via calculation of *Covariance Matrix* applied to attributes, that are our independent predictive variables, used to represent data points.
- Lastly, it is used the technique known as *Singular Valued Decomposition* applied to the overall data points within our dataset.

Reading scikit-learn documentation, I discovered that PCA's derivation uses the *LAPACK implementation* of the *full SVD* or a *randomized truncated SVD* by the method of *Halko et al. 2009*, depending on the shape of the input data and the number of components to extract. Therefore I will descrive mainly that way of deriving the method with respect to the others that, instead, will be described more briefly and roughly.

#### PCA's Iterative based Method
Going in order, as depicted briefly above, I start describing PCA obtained by means of iterative based procedure to extract one at a time a new principal componet explointing the data points at hand.

We begin, recalling that, PCA is defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by some scalar projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on.

We suppose to deal with a data matrix X, with column-wise zero empirical mean, where each of the n rows represents a different repetition of the experiment, and each of the p columns gives a particular kind of feature.

From a math poitn of view, the transformation is defined by a set of p-dimensional vectors of weights or coefficients $\mathbf {w} _{(k)}=(w_{1},\dots ,w_{p})_{(k)}$ that map each row vector $\mathbf{x}_{(i)}$ of X to a new vector of principal component scores ${\displaystyle \mathbf {t} _{(i)}=(t_{1},\dots ,t_{l})_{(i)}}$, given by: ${\displaystyle {t_{k}}_{(i)}=\mathbf {x} _{(i)}\cdot \mathbf {w} _{(k)}\qquad \mathrm {for} \qquad i=1,\dots ,n\qquad k=1,\dots ,l}$.

In this way all the individual variables ${\displaystyle t_{1},\dots ,t_{l}}$ of t considered over the data set successively inherit the maximum possible variance from X, with each coefficient vector w constrained to be a unit vector.

More precisely, the first component In order to maximize variance has to satisfy the following expression:

${\displaystyle \mathbf {w} _{(1)}={\underset {\Vert \mathbf {w} \Vert =1}{\operatorname {\arg \,max} }}\,\left\{\sum _{i}\left(t_{1}\right)_{(i)}^{2}\right\}={\underset {\Vert \mathbf {w} \Vert =1}{\operatorname {\arg \,max} }}\,\left\{\sum _{i}\left(\mathbf {x} _{(i)}\cdot \mathbf {w} \right)^{2}\right\}}$

So, with $w_{1}$ found, the first principal component of a data vector $x_{1}$ can then be given as a score $t_{1(i)} = x_{1} ⋅ w_{1}$ in the transformed co-ordinates, or as the corresponding vector in the original variables, $(x_{1} ⋅ w_{1})w_{1}$.

The others remainig components are computed as folloes. The kth component can be found by subtracting the first k − 1 principal components from X, as in the following expression:

- ${\displaystyle \mathbf {\hat {X}} _{k}=\mathbf {X} -\sum _{s=1}^{k-1}\mathbf {X} \mathbf {w} _{(s)}\mathbf {w} _{(s)}^{\rm {T}}}$

- and then finding the weight vector which extracts the maximum variance from this new data matrix  ${\mathbf {w}}_{{(k)}}={\underset {\Vert {\mathbf {w}}\Vert =1}{\operatorname {arg\,max}}}\left\{\Vert {\mathbf {{\hat {X}}}}_{{k}}{\mathbf {w}}\Vert ^{2}\right\}={\operatorname {\arg \,max}}\,\left\{{\tfrac {{\mathbf {w}}^{T}{\mathbf {{\hat {X}}}}_{{k}}^{T}{\mathbf {{\hat {X}}}}_{{k}}{\mathbf {w}}}{{\mathbf {w}}^{T}{\mathbf {w}}}}\right\}$

It turns out that:
- from the formulas depicted above me get the remaining eigenvectors of $X^{T}X$, with the maximum values for the quantity in brackets given by their corresponding eigenvalues. Thus the weight vectors are eigenvectors of $X^{T}X$.
- The kth principal component of a data vector $x_(i)$ can therefore be given as a score $t_{k(i)} = x_{(i)} ⋅ w_(k)$ in the transformed co-ordinates, or as the corresponding vector in the space of the original variables, $(x_{(i)} ⋅ w_{(k)}) w_{(k)}$, where $w_{(k)}$ is the kth eigenvector of $X^{T}X$.
- The full principal components decomposition of X can therefore be given as: ${\displaystyle \mathbf {T} =\mathbf {X} \mathbf {W}}$, where W is a p-by-p matrix of weights whose columns are the eigenvectors of $X^{T}X$.

#### Covariance Matrix for PCA analysis

PCA made from covarian matrix computation requires the calculation of sample covariance matrix of the dataset as follows: $\mathbf{Q} \propto \mathbf{X}^T \mathbf{X} = \mathbf{W} \mathbf{\Lambda} \mathbf{W}^T$.

The empirical covariance matrix between the principal components becomes ${\displaystyle \mathbf {W} ^{T}\mathbf {Q} \mathbf {W} \propto \mathbf {W} ^{T}\mathbf {W} \,\mathbf {\Lambda } \,\mathbf {W} ^{T}\mathbf {W} =\mathbf {\Lambda } }$.

#### Singular Value Decomposition for PCA analysis

Finally, the principal components transformation can also be associated with another matrix factorization, the singular value decomposition (SVD) of X, ${\displaystyle \mathbf {X} =\mathbf {U} \mathbf {\Sigma } \mathbf {W} ^{T}}$, where more precisely:
- Σ is an n-by-p rectangular diagonal matrix of positive numbers $σ_{(k)}$, called the singular values of X;
- instead U is an n-by-n matrix, the columns of which are orthogonal unit vectors of length n called the left singular vectors of X;
- Then, W is a p-by-p whose columns are orthogonal unit vectors of length p and called the right singular vectors of X.

factorizingn the matrix ${X^{T}X}$, it can be written  as:

${\begin{aligned}\mathbf {X} ^{T}\mathbf {X} &=\mathbf {W} \mathbf {\Sigma } ^{T}\mathbf {U} ^{T}\mathbf {U} \mathbf {\Sigma } \mathbf {W} ^{T}\\&=\mathbf {W} \mathbf {\Sigma } ^{T}\mathbf {\Sigma } \mathbf {W} ^{T}\\&=\mathbf {W} \mathbf {\hat {\Sigma }} ^{2}\mathbf {W} ^{T}\end{aligned}}$

Where we recall that ${\displaystyle \mathbf {\hat {\Sigma }} }$ is the square diagonal matrix with the singular values of X and the excess zeros chopped off that satisfies ${\displaystyle \mathbf {{\hat {\Sigma }}^{2}} =\mathbf {\Sigma } ^{T}\mathbf {\Sigma } } {\displaystyle \mathbf {{\hat {\Sigma }}^{2}} =\mathbf {\Sigma } ^{T}\mathbf {\Sigma } }$. Comparison with the eigenvector factorization of $X^{T}X$ establishes that the right singular vectors W of X are equivalent to the eigenvectors of $X^{T}X$ , while the singular values $σ_{(k)}$ of X are equal to the square-root of the eigenvalues $λ_{(k)}$ of $X^{T}X$ .

At this point we understand that using the singular value decomposition the score matrix T can be written as:

$\begin{align} \mathbf{T} & = \mathbf{X} \mathbf{W} \\ & = \mathbf{U}\mathbf{\Sigma}\mathbf{W}^T \mathbf{W} \\ & = \mathbf{U}\mathbf{\Sigma} \end{align}$

so each column of T is given by one of the left singular vectors of X multiplied by the corresponding singular value. This form is also the polar decomposition of T.

Efficient algorithms exist to calculate the SVD, as in scikit-learn package, of X without having to form the matrix $X^{T}X$, so computing the SVD is now the standard way to calculate a principal components analysis from a data matrix

```python
n_components = rescaledX.shape[1]
pca = PCA(n_components=n_components)
# pca = PCA(n_components=2)

# X_pca = pca.fit_transform(X)
pca = pca.fit(rescaledX)
X_pca = pca.transform(rescaledX)
```

```python
print(f"Cumulative varation explained(percentage) up to given number of pcs:")

tmp_data = []
principal_components = [pc for pc in '2,5,6,7,8,9,10'.split(',')]
for _, pc in enumerate(principal_components):
    n_components = int(pc)

    cum_var_exp_up_to_n_pcs = np.cumsum(pca.explained_variance_ratio_)[n_components-1]
    # print(f"Cumulative varation explained up to {n_components} pcs = {cum_var_exp_up_to_n_pcs}")
    # print(f"# pcs {n_components}: {cum_var_exp_up_to_n_pcs*100:.2f}%")
    tmp_data.append([n_components, cum_var_exp_up_to_n_pcs * 100])

tmp_df = pd.DataFrame(data=tmp_data, columns=['# PCS', 'Cumulative Varation Explained (percentage)'])
tmp_df.head(len(tmp_data))
```

```python
n_components = rescaledX.shape[1]
pca = PCA(n_components=n_components)
# pca = PCA(n_components=2)

#X_pca = pca.fit_transform(X)
pca = pca.fit(rescaledX)
X_pca = pca.transform(rescaledX)

fig = show_cum_variance_vs_components(pca, n_components)

# py.sign_in('franec94', 'QbLNKpC0EZB0kol0aL2Z')
# py.iplot(fig, filename='selecting-principal-components {}'.format(scaler_method))
```

#### Major Pros & Cons of PCA

### Improvements and Conclusions

### References
- Data Domain Information part:
    - (Deck) https://en.wikipedia.org/wiki/Deck_(bridge)
    - (Cantilever bridge) https://en.wikipedia.org/wiki/Cantilever_bridge
    - (Arch bridge) https://en.wikipedia.org/wiki/Deck_(bridge)
- Machine Learning part:
    - (Theory Book) https://jakevdp.github.io/PythonDataScienceHandbook/
    - (Decsion Trees) https://scikit-learn.org/stable/modules/tree.html#tree
    - (SVM) https://scikit-learn.org/stable/modules/svm.html
    - (PCA) https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html
- Chart part:
    - (Seaborn Charts) https://acadgild.com/blog/data-visualization-using-matplotlib-and-seaborn
- Markdown Math part:
    - https://share.cocalc.com/share/b4a30ed038ee41d868dad094193ac462ccd228e2/Homework%20/HW%201.2%20-%20Markdown%20and%20LaTeX%20Cheatsheet.ipynb?viewer=share
    - https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html

```python

```

pittsburgh-bridges-data-set-analysis/backup/Data Space Report (Official) - First Part Descriptive Statistics.ipynb

pittsburgh-bridges-data-set-analysis/models-analyses/descriptive_statistics_analyses/Data Space Report (Official) - First Part Descriptive Statistics.ipynb


# Lambda School Data Science Module 122
## Sampling, Confidence Intervals, and Hypothesis Testing

## Recap: t-tests

```python
from scipy.stats import ttest_ind, ttest_1samp
import numpy as np
import pandas as np
import matplotlib.pyplot as plt
```

```python
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/datasets/master/adult.csv', na_values=" ?")
print(df.shape)
df.head()
```

    (32561, 15)

      age
      workclass
      fnlwgt
      education
      education-num
      marital-status
      occupation
      relationship
      race
      sex
      capital-gain
      capital-loss
      hours-per-week
      country
      salary

      0
      39
      State-gov
      77516
      Bachelors
      13
      Never-married
      Adm-clerical
      Not-in-family
      White
      Male
      2174
      0
      40
      United-States
      <=50K

      1
      50
      Self-emp-not-inc
      83311
      Bachelors
      13
      Married-civ-spouse
      Exec-managerial
      Husband
      White
      Male
      0
      0
      13
      United-States
      <=50K

      2
      38
      Private
      215646
      HS-grad
      9
      Divorced
      Handlers-cleaners
      Not-in-family
      White
      Male
      0
      0
      40
      United-States
      <=50K

      3
      53
      Private
      234721
      11th
      7
      Married-civ-spouse
      Handlers-cleaners
      Husband
      Black
      Male
      0
      0
      40
      United-States
      <=50K

      4
      28
      Private
      338409
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Wife
      Black
      Female
      0
      0
      40
      Cuba
      <=50K

```python
# Remove null values as these will throw off the ttest
df.isnull().sum()
```

    age                  0
    workclass         1836
    fnlwgt               0
    education            0
    education-num        0
    marital-status       0
    occupation        1843
    relationship         0
    race                 0
    sex                  0
    capital-gain         0
    capital-loss         0
    hours-per-week       0
    country            583
    salary               0
    dtype: int64

```python
# salary values
df['salary'].value_counts()
df['salary'].value_counts().index[0]
```

    ' <=50K'

```python
## what is the average age?
popmean=df['age'].mean()
print(popmean)
```

    38.58164675532078

```python
# restrict to high earners
hibucks = df[df['salary']==' >50K']

hibucks.head()
```

      age
      workclass
      fnlwgt
      education
      education-num
      marital-status
      occupation
      relationship
      race
      sex
      capital-gain
      capital-loss
      hours-per-week
      country
      salary

      7
      52
      Self-emp-not-inc
      209642
      HS-grad
      9
      Married-civ-spouse
      Exec-managerial
      Husband
      White
      Male
      0
      0
      45
      United-States
      >50K

      8
      31
      Private
      45781
      Masters
      14
      Never-married
      Prof-specialty
      Not-in-family
      White
      Female
      14084
      0
      50
      United-States
      >50K

      9
      42
      Private
      159449
      Bachelors
      13
      Married-civ-spouse
      Exec-managerial
      Husband
      White
      Male
      5178
      0
      40
      United-States
      >50K

      10
      37
      Private
      280464
      Some-college
      10
      Married-civ-spouse
      Exec-managerial
      Husband
      Black
      Male
      0
      0
      80
      United-States
      >50K

      11
      30
      State-gov
      141297
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Husband
      Asian-Pac-Islander
      Male
      0
      0
      40
      India
      >50K

```python
# is age continuous?
df['age'].describe()
```

    count    32561.000000
    mean        38.581647
    std         13.640433
    min         17.000000
    25%         28.000000
    50%         37.000000
    75%         48.000000
    max         90.000000
    Name: age, dtype: float64

```python
# is the average age of high earners above the population average? yes.
ttest_1samp(hibucks['age'], popmean, nan_policy='omit')
```

    Ttest_1sampResult(statistic=47.71498812387393, pvalue=0.0)

```python
# is there a sig difference between the age of high and low earners? yes.
lobucks = df[df['salary']==' <=50K']
ttest_ind(hibucks['age'], lobucks['age'], nan_policy='omit')
```

    Ttest_indResult(statistic=43.436244240451096, pvalue=0.0)

```python
# subset male and female low earners
lobucks_male = df[(df['salary']==' <=50K') & (df['sex']==' Male')]
lobucks_female = df[(df['salary']==' <=50K') & (df['sex']==' Female')]
df['sex'].value_counts().index[0] # note the trailing space.
```

    ' Male'

```python
# is there a sig difference between the male and female low earners? yes.
ttest_ind(lobucks_female['age'], lobucks_male['age'], nan_policy='omit')
```

    Ttest_indResult(statistic=-5.118769813761817, pvalue=3.098380651743999e-07)

## Prepare - examine other available hypothesis tests

If you had to pick a single hypothesis test in your toolbox, t-test would probably be the best choice - but the good news is you don't have to pick just one! Here's some of the others to be aware of:

```python
# consider a crosstab. Are men and women equally distributed by salary?
gap=pd.crosstab(df['sex'], df['salary'])
gap
```

      salary
      <=50K
      >50K

      sex

      Female
      9592
      1179

      Male
      15128
      6662

```python
# can we test that?
chisquare(gap, axis=None)
```

    Power_divergenceResult(statistic=12478.787844353674, pvalue=0.0)

```python
from scipy.stats import chisquare  # One-way chi square test

# Chi square can take any crosstab/table and test the independence of rows/cols
# The null hypothesis is that the rows/cols are independent -> low chi square
# The alternative is that there is a dependence -> high chi square
# Be aware! Chi square does *not* tell you direction/causation

ind_obs = np.array([[1, 1], [2, 2]]).T
print(ind_obs)
print(chisquare(ind_obs, axis=None))
```

    [[1 2]
     [1 2]]
    Power_divergenceResult(statistic=0.6666666666666666, pvalue=0.8810148425137847)

```python
dep_obs = np.array([[16, 18, 16, 14, 12, 12], [32, 24, 16, 28, 20, 24]]).T
print(dep_obs)
print(chisquare(dep_obs, axis=None))
```

    [[16 32]
     [18 24]
     [16 16]
     [14 28]
     [12 20]
     [12 24]]
    Power_divergenceResult(statistic=23.31034482758621, pvalue=0.015975692534127565)

```python
# Is a distribution normal? This is a poisson distribution
df['age'].plot(kind='hist', bins=20);
```

```python
# Is it normal?
from scipy.stats import normaltest
normaltest(df['age'])
```

    NormaltestResult(statistic=1526.0108462557214, pvalue=0.0)

```python
sample = np.random.normal(5, 25, 1000) # mean, std dev, number of obs
print(normaltest(sample))  # Pretty clearly  normal
```

    NormaltestResult(statistic=0.5266442909770073, pvalue=0.7684942904619738)

```python
# display that!
count, bins, ignored = plt.hist(sample, 14, density=True)
plt.show()
```

```python
# Poisson models arrival times and is related to the binomial (coinflip)
sample = np.random.poisson(5, 1000)
print(normaltest(sample))  # Pretty clearly not normal
```

    NormaltestResult(statistic=55.415631888459714, pvalue=9.260798394897575e-13)

```python
# display that!
count, bins, ignored = plt.hist(sample, 14, density=True)
plt.show()
```

And there's many more! `scipy.stats` is fairly comprehensive, though there are even more available if you delve into the extended world of statistics packages. As tests get increasingly obscure and specialized, the importance of knowing them by heart becomes small - but being able to look them up and figure them out when they *are* relevant is still important.

## Central Limit Theorem

```python
# Flip a coin.
np.random.binomial(n=1, p=.5)
```

    0

```python
# Let me flip a coin twelve times.
coinflips = np.random.binomial(n=1, p=.5, size=12)
coinflips
```

    array([1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1])

```python
# show that.
plt.hist(coinflips);
```

```python
# what is the mean of this sample of coinflips?
coinflips.mean()
```

    0.4166666666666667

```python
# what if I do it again?
coinflips = np.random.binomial(n=1, p=.5, size=12)
coinflips
```

    array([0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1])

```python
# and the mean?
coinflips.mean()
```

    0.3333333333333333

```python
# why are these means different? shouldn't it be the same at .5 for a fair coin?
coinflips = np.random.binomial(n=1, p=.5, size=1000)
coinflips.mean()
```

    0.504

**each of these 'samples' is an estimate of the distribution.** .
https://www.youtube.com/watch?v=Kq7e6cj2nDw

```python
# take 3000 samples of size 12, and make a list of their means.

sample_means = []
for x in range(0,3000):
    coinflips = np.random.binomial(n=1, p=.5, size=12)
    one_sample = coinflips
    sample_means.append(coinflips.mean())

print(len(sample_means))
```

    3000

```python
# look at the first 10 means.
print(sample_means[:10])
```

    [0.5, 0.4166666666666667, 0.3333333333333333, 0.6666666666666666, 0.8333333333333334, 0.5833333333333334, 0.3333333333333333, 0.25, 0.6666666666666666, 0.6666666666666666]

```python
# make a df of one sample
df = pd.DataFrame({'single_sample': one_sample})
df.head()
```

      single_sample

      0
      0

      1
      1

      2
      1

      3
      0

      4
      0

```python
# this df is clearly a binomial distribution (one and zero are the only options)
df.single_sample.hist();
```

```python
# now plot all 3000 sample means
ax = plt.hist(sample_means, bins=13)
plt.title('Distribution of 3000 sample means \n (of 12 coinflips each)');
```

```python
# is this a normal distribution?
normaltest(sample_means)
```

    NormaltestResult(statistic=5.26667716275021, pvalue=0.07183822367832532)

What does the Central Limit Theorem State? That no matter the initial distribution of the population, the distribution of sample means taken will approximate a normal distribution as $n \rightarrow \infty$.

This has very important implications for hypothesis testing and is precisely the reason why the t-distribution begins to approximate the normal distribution as our sample size increases.

```python
# let's pretend each sample included 100 coin flips, not just 12.
sample_means = []
for x in range(0,3000):
    coinflips = np.random.binomial(n=1, p=.5, size=100)
    one_sample = coinflips
    sample_means.append(coinflips.mean())
# the higher my sample size, the closer I get to .5!
print(len(sample_means))
print(sample_means[:10])
```

    3000
    [0.61, 0.41, 0.52, 0.52, 0.52, 0.48, 0.6, 0.57, 0.5, 0.53]

```python
# when I plot, it gets a little taller and a little narrower!
ax = plt.hist(sample_means, bins=13)
plt.title('Distribution of 3000 sample means \n (of 100 coinflips each)');
```

This outcome is what we call the Central Limit Theorem. No matter the distribution of my underlying data, the distribution of the estimates will be distributed normally.

```python
# Compare sets of small and large samples
sample_means_small = []
sample_means_large = []
for x in range(0,3000):
    coinflips_small = np.random.binomial(n=1, p=.5, size=20)
    coinflips_large = np.random.binomial(n=1, p=.5, size=100)
    one_small_sample = coinflips_small
    one_small_large = coinflips_large
    sample_means_small.append(coinflips_small.mean())
    sample_means_large.append(coinflips_large.mean())

print(len(sample_means_small))
print(sample_means_small[:10])
```

    3000
    [0.25, 0.55, 0.55, 0.4, 0.45, 0.55, 0.65, 0.5, 0.35, 0.55]

```python
# Plot that for comparison: it gets taller, narrower, and more normal.
import seaborn as sns
fix, ax = plt.subplots()
for sample in [sample_means_small, sample_means_large]:
    sns.distplot(sample)
```

## Confidence Interval

With 95% confidence, the population mean is within these bounds. This is the [margin of error](https://www.monmouth.edu/polling-institute/reports/monmouthpoll_ia_111219/) we often seen in political polls.

Between the upper and lower bounds we can find 95% of the sample means.

s = standard deviation of the sample
n = sample size
t = test statistic
X = sample mean

As sample size goes up, the standard error (i.e., the spread of our error) gets tighter.

```python
# bring in the stats module (for the t-statistic)
import scipy.stats as stats
```

    0.5004366830554041
    [0 1 0 0 1 0 1 0 0 0 0 1 0 1 1 0 1 0 1 1]

```python
def confidence_interval(data, confidence=0.95):
    """
    Calculate a confidence interval around a sample mean for given data.
    Using t-distribution and two-tailed test, default 95% confidence.

    Arguments:
    data - iterable (list or numpy array) of sample observations
    confidence - level of confidence for the interval

    Returns:
    tuple of (mean, lower bound, upper bound)
    """
    data = np.array(data)
    mean = np.mean(data)
    n = len(data)
    stderr = np.std(data, ddof=1)/np.sqrt(n)
    moe = stderr * stats.t.ppf((1 + confidence) / 2.0, n - 1) # just looking up the corresponding t-stat.
    return (mean, mean - moe, mean + moe)
```

```python
# what's the conf interval for a sample of 500 coin flips?
coinflips = np.random.binomial(n=1, p=.5, size=500)
confidence_interval(coinflips) # mean, upper bound, lower bound.
```

    (0.538, 0.49415050513789005, 0.58184949486211)

```python
# try this again. It should capture the mean 95% of the time.
coinflips = np.random.binomial(n=1, p=.5, size=500)
confidence_interval(coinflips)
```

    (0.486, 0.44204055878984455, 0.5299594412101554)

We want the t-statistic that corresponds to the probability cut off for a given t distribution  (a distribution with a certain 'degrees of freedom').
`stats.t.ppf` is just looking up the corresponding t-statistic.

**Degrees of freedom**
degrees of freedom equals the number of "observations" minus the number of required relations among the observations (e.g., the number of parameter estimates). For a 1-sample t-test, one degree of freedom is spent estimating the mean, and the remaining n - 1 degrees of freedom estimate variability.
https://blog.minitab.com/blog/statistics-and-quality-data-analysis/what-are-degrees-of-freedom-in-statistics

**Looking at stats.t.ppf**

```python
# stats.t.ppf(# probability cutoff, # degrees of freedom)

# 95% confidence level -> .025

# 1 - confidence_level == .05 / 2 -> .025

confidence_level = .95
dof = 42-1

stats.t.ppf((1 + confidence_level) / 2, dof)
```

    2.0195409639828936

## Graphically Represent a Confidence Interval

```python
import matplotlib.pyplot as plt
import seaborn as sns
```

```python
# fake data - one samle of 42 coin flips
coinflips_42 = np.random.binomial(n=1, p=.5, size=42)
# kernel density estimate
sns.kdeplot(coinflips_42)
CI = confidence_interval(coinflips_42)
plt.axvline(x=CI[1], color='red')
plt.axvline(x=CI[2], color='red')
plt.axvline(x=CI[0], color='k');
```

```python
# fake data - 50 samples of 12 coin flips
sample_means = []
for x in range(0,50):
    coinflips = np.random.binomial(n=1, p=.5, size=12)
    one_sample = coinflips
    sample_means.append(coinflips.mean())
# kernel density estimate
sns.kdeplot(sample_means)
CI = confidence_interval(sample_means)
plt.axvline(x=CI[1], color='red')
plt.axvline(x=CI[2], color='red')
plt.axvline(x=CI[0], color='k');
```

## Relationship between Confidence Intervals and T-tests

Confidence Interval == Bounds of statistical significance for our t-test

A sample mean that falls inside of our confidence interval will "FAIL TO REJECT" our null hypothesis

A sample mean that falls outside of our confidence interval will "REJECT" our null hypothesis

```python
from scipy.stats import t, ttest_1samp
```

**Note that scipy has a built-in function to calculate the confidence interval**
t.interval

```python
import numpy as np

coinflip_means = []
for x in range(0,100):
    coinflips = np.random.binomial(n=1, p=.5, size=30)
    coinflip_means.append(coinflips.mean())

print(coinflip_means[:10])
```

    [0.6333333333333333, 0.4666666666666667, 0.4, 0.43333333333333335, 0.4666666666666667, 0.43333333333333335, 0.36666666666666664, 0.5333333333333333, 0.5333333333333333, 0.4666666666666667]

**Rebuild the earlier function**
but using `t.interval` instead.

```python
# Sample Size
n = len(coinflip_means)
# Degrees of Freedom
dof = n-1
# The Mean of Means:
mean = np.mean(coinflip_means)
# Sample Standard Deviation
sample_std = np.std(coinflip_means, ddof=1)
# Standard Error
std_err = sample_std/n**.5

CI = t.interval(.95, dof, loc=mean, scale=std_err)
print("95% Confidence Interval: ", CI)
```

    95% Confidence Interval:  (0.4655337481449764, 0.5011329185216903)

```python
'''You can roll your own CI calculation pretty easily.
The only thing that's a little bit challenging
is understanding the t stat lookup'''

# 95% confidence interval
t_stat = t.ppf(.975, dof)
print("t Statistic:", t_stat)
CI = (mean-(t_stat*std_err), mean+(t_stat*std_err))
print("Confidence Interval", CI)
```

    t Statistic: 1.9842169515086827
    Confidence Interval (0.4655337481449764, 0.5011329185216903)

A null hypothesis that's just inside of our confidence interval == fail to reject

```python
# pick a value for my null hypothesis and check the p-value
ttest_1samp(coinflip_means, .49)
```

    Ttest_1sampResult(statistic=1.0250178821525804, pvalue=0.3078528205334423)

```python
# what if I pick a null hypothesis that's just outside of our confidence interval?
ttest_1samp(coinflip_means, .4818927)
```

    Ttest_1sampResult(statistic=1.948365379483213, pvalue=0.054202611352935866)

```python
# I would only see this 5% of the time, so it's unlikely and I reject the null hypothesis.
# A comparison of what we EXPECT and what we OBSERVE.
```

## Chi Squared Test $\chi^{2}$

A test for dependence of categorical variables. (ie, are they correlated?)

```python
# read back in the earlier data.
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/datasets/master/adult.csv', na_values=" ?")
print(df.shape)
df.head()
```

    (32561, 15)

      age
      workclass
      fnlwgt
      education
      education-num
      marital-status
      occupation
      relationship
      race
      sex
      capital-gain
      capital-loss
      hours-per-week
      country
      salary

      0
      39
      State-gov
      77516
      Bachelors
      13
      Never-married
      Adm-clerical
      Not-in-family
      White
      Male
      2174
      0
      40
      United-States
      <=50K

      1
      50
      Self-emp-not-inc
      83311
      Bachelors
      13
      Married-civ-spouse
      Exec-managerial
      Husband
      White
      Male
      0
      0
      13
      United-States
      <=50K

      2
      38
      Private
      215646
      HS-grad
      9
      Divorced
      Handlers-cleaners
      Not-in-family
      White
      Male
      0
      0
      40
      United-States
      <=50K

      3
      53
      Private
      234721
      11th
      7
      Married-civ-spouse
      Handlers-cleaners
      Husband
      Black
      Male
      0
      0
      40
      United-States
      <=50K

      4
      28
      Private
      338409
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Wife
      Black
      Female
      0
      0
      40
      Cuba
      <=50K

```python
# just the numeric columns
df.describe
```

```python
# just the non-numeric columns (ie, categorical)
df.describe(exclude='number')
```

      workclass
      education
      marital-status
      occupation
      relationship
      race
      sex
      country
      salary

      count
      30725
      32561
      32561
      30718
      32561
      32561
      32561
      31978
      32561

      unique
      8
      16
      7
      14
      6
      5
      2
      41
      2

      top
      Private
      HS-grad
      Married-civ-spouse
      Prof-specialty
      Husband
      White
      Male
      United-States
      <=50K

      freq
      22696
      10501
      14976
      4140
      13193
      27816
      21790
      29170
      24720

```python
# create some binned categories
cut_points = [0, 9, 19, 29, 39, 49, 1000]
label_names = ['0-9', '10-19', '20-29', '30-39', '40-49', '50+']
df['hours_per_week_categories'] = pd.cut(df['hours-per-week'], cut_points, labels=label_names)

df.head()
```

      age
      workclass
      fnlwgt
      education
      education-num
      marital-status
      occupation
      relationship
      race
      sex
      capital-gain
      capital-loss
      hours-per-week
      country
      salary
      hours_per_week_categories

      0
      39
      State-gov
      77516
      Bachelors
      13
      Never-married
      Adm-clerical
      Not-in-family
      White
      Male
      2174
      0
      40
      United-States
      <=50K
      40-49

      1
      50
      Self-emp-not-inc
      83311
      Bachelors
      13
      Married-civ-spouse
      Exec-managerial
      Husband
      White
      Male
      0
      0
      13
      United-States
      <=50K
      10-19

      2
      38
      Private
      215646
      HS-grad
      9
      Divorced
      Handlers-cleaners
      Not-in-family
      White
      Male
      0
      0
      40
      United-States
      <=50K
      40-49

      3
      53
      Private
      234721
      11th
      7
      Married-civ-spouse
      Handlers-cleaners
      Husband
      Black
      Male
      0
      0
      40
      United-States
      <=50K
      40-49

      4
      28
      Private
      338409
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Wife
      Black
      Female
      0
      0
      40
      Cuba
      <=50K
      40-49

```python
# sex
df['sex'].value_counts()
```

     Male      21790
     Female    10771
    Name: sex, dtype: int64

```python
# binned category
df['hours_per_week_categories'].value_counts()
```

    40-49    18336
    50+       6462
    30-39     3667
    20-29     2392
    10-19     1246
    0-9        458
    Name: hours_per_week_categories, dtype: int64

```python
df = df.sort_values(by='hours_per_week_categories', ascending=True)

df.head()
```

      age
      workclass
      fnlwgt
      education
      education-num
      marital-status
      occupation
      relationship
      race
      sex
      capital-gain
      capital-loss
      hours-per-week
      country
      salary
      hours_per_week_categories

      31290
      55
      Self-emp-not-inc
      41938
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Wife
      White
      Female
      0
      0
      8
      United-States
      <=50K
      0-9

      5172
      32
      NaN
      134886
      HS-grad
      9
      Married-civ-spouse
      NaN
      Wife
      White
      Female
      0
      0
      2
      United-States
      >50K
      0-9

      22928
      17
      NaN
      332666
      10th
      6
      Never-married
      NaN
      Own-child
      White
      Female
      0
      0
      4
      United-States
      <=50K
      0-9

      7902
      35
      Private
      359131
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Wife
      White
      Female
      7298
      0
      8
      NaN
      >50K
      0-9

      6604
      41
      Private
      406603
      HS-grad
      9
      Never-married
      Other-service
      Not-in-family
      White
      Male
      0
      0
      6
      Iran
      <=50K
      0-9

```python
contingency_table = pd.crosstab(df['sex'], df['hours_per_week_categories'], margins=True)

contingency_table
```

      hours_per_week_categories
      0-9
      10-19
      20-29
      30-39
      40-49
      50+
      All

      sex

      Female
      235
      671
      1287
      1914
      5636
      1028
      10771

      Male
      223
      575
      1105
      1753
      12700
      5434
      21790

      All
      458
      1246
      2392
      3667
      18336
      6462
      32561

```python
# create a set of female counts
femalecount = contingency_table.iloc[0][0:6].values
femalecount
```

    array([ 235,  671, 1287, 1914, 5636, 1028])

```python
# create a set of male counts
malecount = contingency_table.iloc[1][0:6].values
malecount
```

    array([  223,   575,  1105,  1753, 12700,  5434])

```python
#Plots the bar chart
fig = plt.figure(figsize=(10, 5))
sns.set(font_scale=1.8)
categories = ["0-9","10-19","20-29","30-39","40-49","50+"]
p1 = plt.bar(categories, malecount, 0.55, color='#d62728')
p2 = plt.bar(categories, femalecount, 0.55, bottom=malecount)
plt.legend((p2[0], p1[0]), ('Female', 'Male'))
plt.xlabel('Hours per Week Worked')
plt.ylabel('Count')
plt.show()
```

## Expected Value Calculation
\begin{align}
expected_{i,j} =\frac{(row_{i} \text{total})(column_{j} \text{total}) }{(\text{total observations})}
\end{align}

```python
row_sums = contingency_table.iloc[0:2, 6].values
col_sums = contingency_table.iloc[2, 0:6].values

print(row_sums)
print(col_sums)
```

    [10771 21790]
    [  458  1246  2392  3667 18336  6462]

```python
total = contingency_table.loc['All','All']
total
```

    32561

```python
df.shape[0]
```

    32561

```python
expected = []
for i in range(len(row_sums)):
    expected_row = []
    for column in col_sums:
        expected_val = column*row_sums[i]/total
        expected_row.append(expected_val)
    expected.append(expected_row)

expected = np.array(expected)
print(expected.shape)
print(expected)
```

    (2, 6)
    [[  151.50388502   412.16995793   791.26046497  1213.02346365
       6065.44811277  2137.59411566]
     [  306.49611498   833.83004207  1600.73953503  2453.97653635
      12270.55188723  4324.40588434]]

```python
observed = pd.crosstab(df['sex'], df['hours_per_week_categories']).values
print(observed.shape)
observed
```

    (2, 6)

    array([[  235,   671,  1287,  1914,  5636,  1028],
           [  223,   575,  1105,  1753, 12700,  5434]])

## Chi-Squared Statistic with Numpy

\begin{align}
\chi^2 = \sum \frac{(observed_{i}-expected_{i})^2}{(expected_{i})}
\end{align}

For the $observed$ values we will just use a version of our contingency table without the margins as a numpy array. In this way, if our observed values array and our expected values array are the same shape, then we can subtract them and divide them directly which makes the calculations a lot cleaner. No for loops!

```python
# Array broadcasting will work with numpy arrays but not python lists
chi_squared = ((observed - expected)**2/(expected)).sum()
print(f"Chi-Squared: {chi_squared}")
```

    Chi-Squared: 2287.190943926107

```python
# Degrees of Freedom of a Chi-squared test

#degrees_of_freedom = (num_rows - 1)(num_columns - 1)

# Calculate Degrees of Freedom
dof = (len(row_sums)-1)*(len(col_sums)-1)
print(f"Degrees of Freedom: {dof}")
```

    Degrees of Freedom: 5

## Run a $\chi^{2}$ Test using Scipy

```python
chi_squared, p_value, dof, expected = stats.chi2_contingency(observed)

print(f"Chi-Squared: {chi_squared}")
print(f"P-value: {p_value}")
print(f"Degrees of Freedom: {dof}")
print("Expected: \n", np.array(expected))
```

    Chi-Squared: 2287.190943926107
    P-value: 0.0
    Degrees of Freedom: 5
    Expected:
     [[  151.50388502   412.16995793   791.26046497  1213.02346365
       6065.44811277  2137.59411566]
     [  306.49611498   833.83004207  1600.73953503  2453.97653635
      12270.55188723  4324.40588434]]

Null Hypothesis: Hours worked per week bins is **independent** of sex.

Due to a p-value of 0, we REJECT the null hypothesis that hours worked per week and sex are independent, and conclude that there is an association between hours worked per week and sex.

### Another example

```python
# sex and salary
observed=pd.crosstab(df['sex'], df['salary']).values
```

```python
# chi square test
chi_squared, p_value, dof, expected = stats.chi2_contingency(observed)

print(f"Chi-Squared: {chi_squared}")
print(f"P-value: {p_value}")
print(f"Degrees of Freedom: {dof}")
print("Expected: \n", np.array(expected))
```

    Chi-Squared: 1517.813409134445
    P-value: 0.0
    Degrees of Freedom: 1
    Expected:
     [[ 8177.24025675  2593.75974325]
     [16542.75974325  5247.24025675]]

```python
# age and salary
df['age_groups']=0
df.loc[df['age']>20, 'age_groups']=1
df.loc[df['age']>30, 'age_groups']=2
df.loc[df['age']>40, 'age_groups']=3
df.loc[df['age']>50, 'age_groups']=4
df['age_groups'].value_counts()
```

    2    8546
    1    8162
    3    6983
    4    6460
    0    2410
    Name: age_groups, dtype: int64

```python
# crosstab
observed=pd.crosstab(df['age_groups'], df['salary']).values
observed
```

    array([[2408,    2],
           [7482,  680],
           [6140, 2406],
           [4328, 2655],
           [4362, 2098]])

```python
# chi square test
chi_squared, p_value, dof, expected = stats.chi2_contingency(observed)

print(f"Chi-Squared: {chi_squared}")
print(f"P-value: {p_value}")
print(f"Degrees of Freedom: {dof}")
print("Expected: \n", np.array(expected))
```

    Chi-Squared: 2935.441819762824
    P-value: 0.0
    Degrees of Freedom: 4
    Expected:
     [[1829.64896656  580.35103344]
     [6196.51239213 1965.48760787]
     [6488.04152207 2057.95847793]
     [5301.42686035 1681.57313965]
     [4904.3702589  1555.6297411 ]]

```python

```


# `GiRaFFE_NRPy`: Solving the Induction Equation

## Author: Patrick Nelson

Our goal in this module is to write the code necessary to solve the induction equation
$$
\partial_t A_i = \underbrace{\epsilon_{ijk} v^j B^k}_{\rm No\ Gauge\ terms} - \underbrace{\partial_i \left(\alpha \Phi - \beta^j A_j \right)}_{\rm Gauge\ terms}
$$

Since we are not using staggered grids, we can greatly simplify this algorithm with respect to the version used in the original `GiRaFFE`. We turn to Tóth's [paper](https://www.sciencedirect.com/science/article/pii/S0021999100965197?via%3Dihub), Eqs. 30 and 31, and a 3D version of the same algorithm in **TODO: RIT paper**.

Consider the electric field $E_i = \epsilon_{ijk} v^j B^k$ (this relation assumes the ideal MHD limit, which is also assumed in FFE).

Consider the point $i,j,k$. Let components of tensors be indicated with braces, i.e. the $i^{\rm th}$ component of the electric field at point $i,j,k$ will be written as $\left(E_{\{i\}}\right)_{i,j,k}$. Then
\begin{align}
\left(E_{\{1\}}\right)_{i,j,k} = \frac{1}{4}(v^{\{2\}})B^{\{3\}})_{i,j-\tfrac{1}{2},k} &+ \frac{1}{4}(v^{\{2\}})B^{\{3\}})_{i,j+\tfrac{1}{2},k}\\
- \frac{1}{4}(v^{\{3\}})B^{\{2\}})_{i,j,k-\tfrac{1}{2}} &- \frac{1}{4}(v^{\{3\}})B^{\{2\}})_{i,j,k+\tfrac{1}{2}}
\end{align}

The other components follow via a cyclic permutation of the indices. Note a potential complication here: When we are calculating $i^{\rm th}$ component of the electric field, we are concerned with the reconstructed quantities in the $j^{\rm th}$ and $k^{\rm th}$ directions. This means that it will be sensible to do something similar to what we do with the A2B module and think first about the directions in which a stencil goes, and *then* the terms that involve it.

In this case, we will compute the face-value products of $v^i$ and $B^i$ in, say, the 0th direction **TODO: rectify off-by-one above**. Then, we will compute the parts of components of the electric field that depend on those: the 1st and 2nd direction.

An outline of a general finite-volume method is as follows, with the current step in bold:
1. The Reconstruction Step - Piecewise Parabolic Method
    1. Within each cell, fit to a function that conserves the volume in that cell using information from the neighboring cells
        * For PPM, we will naturally use parabolas
    1. Use that fit to define the state at the left and right interface of each cell
    1. Apply a slope limiter to mitigate Gibbs phenomenon
1. Interpolate the value of the metric gridfunctions on the cell faces
1. **Solving the Riemann Problem - Harten, Lax, (This notebook, $E_i$ only)**
    1. **Use the left and right reconstructed states to calculate the unique state at boundary**

We will assume in this notebook that the reconstructed velocities and magnetic fields are available on cell faces as input. We will also assume that the metric gridfunctions have been interpolated on the metric faces.

Solving the Riemann problem, then, consists of two substeps: First, we compute the flux through each face of the cell. Then, we add the average of these fluxes to the right-hand side of the evolution equation for the vector potential.

We begin by importing the NRPy+ core functionality. We also import the Levi-Civita symbol, the GRHD module, and the GRFFE module.

```python
# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

from outputC import *            # NRPy+: Core C code output module
import finite_difference as fin  # NRPy+: Finite difference C code generation module
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import grid as gri               # NRPy+: Functions having to do with numerical grids
import loop as lp                # NRPy+: Generate C code loops
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm   # NRPy+: Reference metric support
import cmdline_helper as cmd     # NRPy+: Multi-platform Python command-line interface
import shutil, os, sys           # Standard Python modules for multiplatform OS-level functions

thismodule = "GiRaFFE_NRPy-Induction_Equation"

import GRHD.equations as GRHD
import GRFFE.equations as GRFFE

# Import the Levi-Civita symbol and build the corresponding tensor.
# We already have a handy function to define the Levi-Civita symbol in WeylScalars
import WeylScal4NRPy.WeylScalars_Cartesian as weyl

```

This function is identical to the one done by Stilde_flux. See [that tutorial](Tutorial-GiRaFFE_NRPy-Stilde-flux.ipynb#hydro_speed) for further information on the derivation.

```python
# We'll write this as a function so that we can calculate the expressions on-demand for any choice of i
def find_cp_cm(lapse,shifti,gupii):
    # Inputs:  u0,vi,lapse,shift,gammadet,gupii
    # Outputs: cplus,cminus

    # a = 1/(alpha^2)
    a = 1/(lapse*lapse)
    # b = 2 beta^i / alpha^2
    b = 2 * shifti /(lapse*lapse)
    # c = -g^{ii} + (beta^i)^2 / alpha^2
    c = - gupii + shifti*shifti/(lapse*lapse)

    # Now, we are free to solve the quadratic equation as usual. We take care to avoid passing a
    # negative value to the sqrt function.
    detm = b*b - 4*a*c
    detm = sp.sqrt(sp.Rational(1,2)*(detm + nrpyAbs(detm)))
    global cplus,cminus
    cplus  = sp.Rational(1,2)*(-b/a + detm/a)
    cminus = sp.Rational(1,2)*(-b/a - detm/a)

```

This function is identical to the one done by Stilde_flux. For more information, see [here](Tutorial-GiRaFFE_NRPy-Stilde-flux.ipynb#fluxes).

```python
# We'll write this as a function, and call it within HLLE_solver, below.
def find_cmax_cmin(flux_dirn,gamma_faceDD,beta_faceU,alpha_face):
    # Inputs:  flux direction flux_dirn, Inverse metric gamma_faceUU, shift beta_faceU,
    #          lapse alpha_face, metric determinant gammadet_face
    # Outputs: maximum and minimum characteristic speeds cmax and cmin
    # First, we need to find the characteristic speeds on each face
    gamma_faceUU,unusedgammaDET = ixp.generic_matrix_inverter3x3(gamma_faceDD)
    find_cp_cm(alpha_face,beta_faceU[flux_dirn],gamma_faceUU[flux_dirn][flux_dirn])
    cpr = cplus
    cmr = cminus
    find_cp_cm(alpha_face,beta_faceU[flux_dirn],gamma_faceUU[flux_dirn][flux_dirn])
    cpl = cplus
    cml = cminus

    # The following algorithms have been verified with random floats:

    global cmax,cmin
    # Now, we need to set cmax to the larger of cpr,cpl, and 0
    cmax = sp.Rational(1,2)*(cpr+cpl+nrpyAbs(cpr-cpl))
    cmax = sp.Rational(1,2)*(cmax+nrpyAbs(cmax))

    # And then, set cmin to the smaller of cmr,cml, and 0
    cmin =  sp.Rational(1,2)*(cmr+cml-nrpyAbs(cmr-cml))
    cmin = -sp.Rational(1,2)*(cmin-nrpyAbs(cmin))

```

See GRHydro paper for equations (TBA)**TODO**

Here, we we calculate the flux and state vectors for the electric field.
The flux vector in the $i^{\rm th}$ direction is given as
$$
F(U) = \epsilon_{ijk} v^j B^k,
$$
where $\epsilon_{ijk} = [ijk]\sqrt{\gamma}$, $[ijk]$ is the Levi-Civita symbol, $\gamma$ is the determinant of the three-metric, and $v^j = \alpha \bar{v^j} - \beta^j$ is the drift velocity; the state vector in the $i^{\rm th}$ direction is $U = B^i$.

**TODO** Why is there only one free index in this flux, but two for $\tilde{S}_i$? I think the LC tensor might be implicitly contain a loop that had to be explicitly written in the other case?

```python
def calculate_flux_and_state_for_Induction(flux_dirn, gammaDD,betaU,alpha,ValenciavU,BU):
    GRHD.compute_sqrtgammaDET(gammaDD)
    LeviCivitaDDD = weyl.define_LeviCivitaSymbol_rank3()
    LeviCivitaDDD = ixp.zerorank3()
    for i in range(DIM):
        for j in range(DIM):
            for k in range(DIM):
                LCijk = LeviCivitaDDD[i][j][k]
                LeviCivitaDDD[i][j][k] = LCijk * GRHD.sqrtgammaDET)
    #             LeviCivitaUUU[i][j][k] = LCijk / sp.sqrt(gammadet)

    global U,F
    # Flux F = \epsilon_{ijk} v^j B^k
    F = sp.sympify(0)
    for j in range(3):
        for k in range(3):
            F = LeviCivitaDDD[flux_dirn][j][k] * (alpha*ValenciavU[j]-betaU[j]) * BU[k]
    # U = B^i
    U = BU[flux_dirn]

```

Now, we write a standard HLLE solver based on eq. 3.15 in [the HLLE paper](https://epubs.siam.org/doi/pdf/10.1137/1025002),
$$
F^{\rm HLL} = \frac{c_{\rm min} f_{\rm R} + c_{\rm max} f_{\rm L} - c_{\rm min} c_{\rm max} ({\rm st\_x\_r}-{\rm st\_x\_l})}{c_{\rm min} + c_{\rm max}}
$$

```python
def HLLE_solver(cmax, cmin, Fr, Fl, Ur, Ul):
    # This solves the Riemann problem for the mom_comp component of the momentum
    # flux StildeD in the flux_dirn direction.

    # st_j_flux = (c_\min f_R + c_\max f_L - c_\min c_\max ( st_j_r - st_j_l )) / (c_\min + c_\max)
    return (cmin*Fr + cmax*Fl - cmin*cmax*(Ur-Ul) )/(cmax + cmin)
```

Here, we will use the function we just wrote to calculate the flux through a face. We will pass the reconstructed Valencia 3-velocity and magnetic field on either side of an interface to this function (designated as the "left" and "right" sides) along with the value of the 3-metric, shift vector, and lapse function on the interface. However, unlike when we used this method to calculate the flux term, the RHS of each component of $A_i$ does not depend on all three of the flux directions. Instead,

```python
def calculate_E_i_flux(inputs_provided=True,alpha_face=None,gamma_faceDD=None,beta_faceU=None,\
                       Valenciav_rU=None,B_rU=None,Valenciav_lU=None,B_lU=None):
    find_cmax_cmin(flux_dirn,gamma_faceDD,beta_faceU,alpha_face)

    global E_fluxD
    # We will need to add to this rhs formula in several functions. To avoid overwriting previously written
    # data, we use declarerank1() instead of zerorank1().
    E_fluxD = ixp.declarerank1("A_rhsD",DIM=3)
    for flux_dirn in range(3):
        calculate_flux_and_state_for_Induction(flux_dirn, gamma_faceDD,beta_faceU,alpha_face,\
                                               Valenciav_rU,B_rU)
        Fr = F
        Ur = U
        calculate_flux_and_state_for_Induction(flux_dirn, gamma_faceDD,beta_faceU,alpha_face,\
                                               Valenciav_lU,B_lU)
        Fl = F
        Ul = U
        E_fluxD[flux_dirn] += HLLE_solver(cmax, cmin, Fr, Fl, Ur, Ul)

```

Below are the gridfunction registrations we will need for testing. We will pass these to the above functions to self-validate the module that corresponds with this tutorial.

```python
# We will pass values of the gridfunction on the cell faces into the function. This requires us
# to declare them as C parameters in NRPy+. We will denote this with the _face infix/suffix.
alpha_face = gri.register_gridfunctions("AUXEVOL","alpha_face")
gamma_faceDD = ixp.register_gridfunctions_for_single_rank2("AUXEVOL","gamma_faceDD","sym01")
beta_faceU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","beta_faceU")

# We'll need some more gridfunctions, now, to represent the reconstructions of BU and ValenciavU
# on the right and left faces
Valenciav_rU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","Valenciav_rU",DIM=3)
B_rU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","B_rU",DIM=3)
Valenciav_lU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","Valenciav_lU",DIM=3)
B_lU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","B_lU",DIM=3)

```