## Subsurface Data Analytics

### Polynomial Regression for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Polynomial Regression for Subsurface Modeling in Python

Here's a simple workflow, demonstration of polynomial regression for subsurface modeling workflows. This should help you get started with building subsurface models that data analytics and machine learning. Here's some basic details about linear regression.

#### Linear Regression

Linear regression for prediction.  Here are some key aspects of linear regression:

**Parametric Model**

* the fit model is a simple weighted linear additive model based on all the available features, $x_1,\ldots,x_m$.

* the model takes the form of $y = \sum_{\alpha = 1}^m b_{\alpha} x_{\alpha} + b_0$

**Least Squares**

* least squares optimization is applied to select the model parameters, $b_1,\ldots,b_m,b_0$

* we minize the error over the trainind data $\sum_{i=1}^n (y_i - (\sum_{\alpha = 1}^m b_{\alpha} x_{\alpha} + b_0))^2$

* this could be simplified as the sum of square error over the training data, $\sum_{i=1}^n (\Delta y_i)^2$

**Assumptions**

* **Error-free** - predictor variables are error free, not random variables
* **Linearity** - response is linear combination of feature(s)
* **Constant Variance** - error in response is constant over predictor(s) value
* **Independence of Error** - error in response are uncorrelated with each other
* **No multicollinearity** - none of the features are redundant with other features

#### Polynomial Regression

It can be shown that polynomial regression is just linear regression applied to a polynomial expansion of the predictor features.

\begin{equation}
X_{j} \rightarrow X_{j}, X_{j}^2, X_{j}^3, \ldots X_{j}^k
\end{equation}

where we have $j = 1, \ldots, m$ original features.

We now have a expanded set of predictor features.

\begin{equation}
h_{j,k}(X_j) = X_j^k
\end{equation}

were we have $j = 1, \ldots, m$ original features and $k = 1, \ldots, K$ polynomial orders.

We can now state our model as a linear regression of the transformed features.

\begin{equation}
y = f(x) = \sum_{j=1}^{m} \sum_{k = 1}^{K} \beta_{j,k} h_{j,m}(X_j)
\end{equation}

So our workflow is:

* apply polynomial basis expansion

* perform linear regression

#### Other Resources

This is a tutorial / demonstration of **Linear Regression**.  In $Python$, the $SciPy$ package, specifically the $Stats$ functions (https://docs.scipy.org/doc/scipy/reference/stats.html) provide excellent tools for efficient use of statistics.
I have previously provided this example in R and posted it on GitHub:

1. R https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.R
2. Rmd with docs https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.Rmd
3. knit as an HTML document(https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.html)

#### Workflow Goals

Learn the basics of time series analysis in Python to for analysis, modeling and prediction with production data. This includes:

* Basic Python workflows and data preparation

* Training / fitting a linear regression model

* Model Checking

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

There are examples below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Import Required Packages

Let's import the GeostatsPy package.

```python
import os                                                   # to set current working directory
import numpy as np                                          # arrays and matrix math
import scipy                                                # hermite polynomials
import scipy.stats as st                                    # statistical methods
import pandas as pd                                         # DataFrames
import matplotlib.pyplot as plt                             # for plotting
from sklearn.linear_model import LinearRegression           # linear regression with scikit learn
from sklearn.preprocessing import PolynomialFeatures        # polynomial basis expansion
from scipy import stats                                     # statistical summary from a 2D ndarray
import seaborn as sns                                       # multivariate statistical displays
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see below) data file in this working directory.

```python
#os.chdir("d:/PGE337")                                      # set the working directory
```

#### Loading Data

Let's load the provided dataset. 'Density_Por_data.csv' is available at https://github.com/GeostatsGuy/GeoDataSets. It is a comma delimited file with 20 density measures ($\frac{g}{cm^3}$) and porosity measures from 2 rock units from the subsurface, porosity (as a fraction). We load it with the pandas 'read_csv' function into a data frame we called 'df' and then preview it by printing a slice and by utilizing the 'head' DataFrame member function (with a nice and clean format, see below).

```python
#df = pd.read_csv("1D_Porosity.csv")                        # read a .csv file in as a DataFrame
df = pd.read_csv(r"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/1D_Porosity.csv") # load the data from Dr. Pyrcz's github repository
df.head()                                                   # preview the data
```

      Depth
      Nporosity

      0
      0.25
      -1.37

      1
      0.50
      -2.08

      2
      0.75
      -1.67

      3
      1.00
      -1.16

      4
      1.25
      -0.24

It is useful to review the summary statistics of our loaded DataFrame.  That can be accomplished with the 'describe' DataFrame member function.  We transpose to switch the axes for ease of visualization.

```python
df.describe().transpose()                                   # summary statistics
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Depth
      40.0
      5.12500
      2.922613
      0.25
      2.6875
      5.125
      7.5625
      10.00

      Nporosity
      40.0
      0.02225
      0.992111
      -2.08
      -0.7575
      0.140
      0.7425
      2.35

Here we extract the X1 and X2 unit porosity samples from the DataFrame into separate arrays called 'X1' and 'X2' for convenience.

```python
Npor = df['Nporosity']                                      # extract the 1D ndarrays from the DataFrame
depth = df['Depth']
```

#### Linear Regression Model

Let's first calculate the linear regression model

```python
lin = LinearRegression()                                        # instantiate linear regression object, note no hyperparameters
lin.fit(depth.values.reshape(-1, 1), Npor)                      # train linear regression
slope = lin.coef_[0]
intercept = lin.intercept_

print('The model parameters are, slope (b1) = ' + str(round(slope,2)) + ', and the intercept (b0) = ' + str(round(intercept,2)))
```

    The model parameters are, slope (b1) = 0.14, and the intercept (b0) = -0.69

Let's plot the data and the model.

```python
depth_values = np.linspace(0.0,10.0,100)
plt.subplot(111)
plt.plot(depth, Npor, 'o', label='sample data', color = 'red', alpha = 0.2, markeredgecolor = 'black')
plt.plot(depth_values, intercept + slope*depth_values, label='model', color = 'black')
plt.title('NPorosity vs Depth')
plt.xlabel('Z (m)')
plt.ylabel('NPorosity')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

#### Comparison to a Non Parametric Model

Let's run a quick decision tree for a comparision to a non parametric model.

* we gain significant flexibility to fit any patterns from the data

* requires more inference as nonparametric is actually parameter rich!

This first model is a decision tree regression model.

```python
from sklearn import tree                                    # tree program from scikit learn (packag
depth_values = np.linspace(0.0,10.0,1000)
my_tree = tree.DecisionTreeRegressor(min_samples_leaf=1, max_depth = 20)
my_tree = my_tree.fit(depth.values.reshape(-1, 1), Npor)
DT_Npor = my_tree.predict(depth_values.reshape(-1,1))
plt.subplot(111)
plt.plot(depth_values, DT_Npor, label='model', color = 'black')
plt.plot(depth, Npor, 'o', label='sample data', color = 'red', alpha = 0.6, markeredgecolor = 'black')
plt.title('Decision Tree Prediction of NPorosity from Depth')
plt.xlabel('Depth (m)')
plt.ylabel('NPorosity')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

and here is a random forest model:

```python
from sklearn.ensemble import RandomForestRegressor          # random forest method
seed = 73093                                                # set the random forest hyperparameters
max_depth = 5
num_tree = 1000
max_features = 1

my_forest = RandomForestRegressor(max_depth=max_depth, random_state=seed,n_estimators=num_tree, max_features=max_features)
my_forest.fit(X = depth.values.reshape(-1, 1), y = Npor)
RF_Npor = my_forest.predict(depth_values.reshape(-1,1))
plt.subplot(111)
plt.plot(depth_values, RF_Npor, label='model', color = 'black')
plt.plot(depth, Npor, 'o', label='sample data', color = 'red', alpha = 0.6, markeredgecolor = 'black')
plt.title('Decision Tree Prediction of NPorosity from Depth')
plt.xlabel('Z (m)')
plt.ylabel('NPorosity')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

#### Standardized Features

Let's work with standardized features. I switched to Gaussian transform (see below).

```python
# from sklearn.preprocessing import StandardScaler

# scaler = StandardScaler()                                  # instantiate the scaler
# sfeatures = scaler.fit_transform(df.values)                # standardize all the values extracted from the DataFrame
# df_st = pd.DataFrame()                                     # instantiate a new DataFrame
# df_st = pd.DataFrame(sfeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
# depth_st = df_st['Depth']
# Npor_st = df_st['Nporosity']
# df_st.head()                                               # preview the the new DataFrame
```

#### Gaussian Anamorphosis

Let's transform the features to standard normal.

* Gaussian distribution
* mean of 0.0
* standard deviation of 1.0

The porosity feature was 'transformed' to Gaussian previously, but there is an opportunity to clean it up.

* compare the original and transformed below

```python
from sklearn.preprocessing import QuantileTransformer
import geostatspy.geostats as geostats

# I would like to use the scikit-learn method, but it creates outliers!
#nscore = QuantileTransformer(n_quantiles=40, random_state=73, output_distribution = 'normal')
#nsfeatures = nscore.fit_transform(df)                                                             # standardize all the values extracted from the DataFrame
#df_ns = pd.DataFrame()                                      # instantiate a new DataFrame
#df_ns = pd.DataFrame(nsfeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame

#Transform to Gaussian with GeostatsPy
df_ns = pd.DataFrame()
df_ns['Nporosity'], tvPor, tnsPor = geostats.nscore(df, 'Nporosity') # nscore transform for all facies porosity
df_ns['Depth'], tvdepth, tnsdepth = geostats.nscore(df, 'Depth')  # nscore transform for all facies permeability
depth_ns = df_ns['Depth']
Npor_ns = df_ns['Nporosity']
df_ns.head()
```

      Nporosity
      Depth

      0
      -1.356312
      -2.026808

      1
      -2.219599
      -1.780464

      2
      -1.780464
      -1.534121

      3
      -1.091620
      -1.356312

      4
      -0.221119
      -1.213340

Let's make some good cumulative distribution function plots to check the original and transformed variables.

* the results look very good

We are doing this because we will need a Gaussian distribution for the predictor feature for orthogonality.  More later!

```python
plt.subplot(221)                                              # plot original sand and shale porosity histograms
plt.hist(df['Depth'], facecolor='red',bins=np.linspace(0.0,10.0,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label='Original')
plt.xlim([0.0,10.0]); plt.ylim([0,1.0])
plt.xlabel('Depth (m)'); plt.ylabel('Frequency'); plt.title('Original Depth')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(222)
plt.hist(df_ns['Depth'], facecolor='blue',bins=np.linspace(-3.0,3.0,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label = 'Trans')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Depth (Standard Normal)'); plt.ylabel('Frequency'); plt.title('Nscore Depth')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(223)                                        # plot nscore transformed sand and shale histograms
plt.hist(df['Nporosity'], facecolor='red',bins=np.linspace(-3.0,3.0,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label='Original')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Porosity (Near Gaussian)'); plt.ylabel('Frequency'); plt.title('Original Porosity')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(224)                                        # plot nscore transformed sand and shale histograms
plt.hist(df_ns['Nporosity'], facecolor='blue',bins=np.linspace(-3.0,3.0,100000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label = 'Trans')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Porosity (Standard Normal)'); plt.ylabel('Frequency'); plt.title('Nscore Porosity')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=2.2, wspace=0.2, hspace=0.3)
plt.show()
```

#### Linear Regression Model with Standardized Features

Let's repeat the linear regression model, now with the standardized features.

```python
lin_ns = LinearRegression()
lin_ns.fit(depth_ns.values.reshape(-1, 1), Npor_ns)
slope_ns = lin_ns.coef_[0]
intercept_ns = lin_ns.intercept_
```

Now visualize the model.

* Quite a poor fit

```python
depth_ns_values = np.linspace(-3.0,3.0,100)
plt.subplot(111)
plt.plot(depth_ns, Npor_ns, 'o', label='sample data', color = 'red', alpha = 0.2, markeredgecolor = 'black')
plt.plot(depth_ns_values, intercept_ns + slope_ns*depth_ns_values, label='model', color = 'black')
plt.title('Standardized Porosity vs Standardized Depth')
plt.xlabel('Standardized Depth')
plt.ylabel('Standardized Porosity')
plt.legend(); plt.xlim(-2,2)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

#### Polynomial Regression

We will do polynomial regression by hand:

* create the polynomial basis expansion of the original predictor feature

* perform linear regression on the polynomial basis expansion

```python
poly4 = PolynomialFeatures(degree = 4)                       # instantiate polynomial expansion
depth_ns_poly4 = poly4.fit_transform(depth_ns.values.reshape(-1, 1))# apply polynomial expansion to transformed predictor feature
df_depth_ns_poly4 = pd.DataFrame({'Values':depth_ns.values,'0th': depth_ns_poly4[:,0],'1st': depth_ns_poly4[:,1],'2nd': depth_ns_poly4[:,2], '3rd': depth_ns_poly4[:,3], '4th': depth_ns_poly4[:,4]}) # make a new DataFrame from the vectors
df_depth_ns_poly4.head()
```

      Values
      0th
      1st
      2nd
      3rd
      4th

      0
      -2.026808
      1.0
      -2.026808
      4.107951
      -8.326029
      16.875264

      1
      -1.780464
      1.0
      -1.780464
      3.170053
      -5.644167
      10.049238

      2
      -1.534121
      1.0
      -1.534121
      2.353526
      -3.610592
      5.539084

      3
      -1.356312
      1.0
      -1.356312
      1.839582
      -2.495046
      3.384060

      4
      -1.213340
      1.0
      -1.213340
      1.472193
      -1.786270
      2.167352

Now let's check the correlation between the polynomial basis expansion of the original predictor features data.

```python
df_depth_ns_poly4.iloc[:,2:].corr()
```

      1st
      2nd
      3rd
      4th

      1st
      1.000000
      0.059005
      0.850982
      0.111841

      2nd
      0.059005
      1.000000
      0.147450
      0.938188

      3rd
      0.850982
      0.147450
      1.000000
      0.243912

      4th
      0.111841
      0.938188
      0.243912
      1.000000

#### Visualize the Polynomial Expansion Features' Pairwise Relationship

```python
sns.pairplot(df_depth_ns_poly4.iloc[:,2:],vars=['1st','2nd','3rd','4th'],markers='o', kind='reg',diag_kind='kde')
```

Let's visualize the polynomial expansion over the standardized depth.

```python
plt.subplot(111)
plt.plot(depth_ns_values, poly4.fit_transform(depth_ns_values.reshape(-1, 1)) [:,0], label='0th', color = 'black')
plt.plot(depth_ns_values, poly4.fit_transform(depth_ns_values.reshape(-1, 1)) [:,1], label='1th', color = 'blue')
plt.plot(depth_ns_values, poly4.fit_transform(depth_ns_values.reshape(-1, 1)) [:,2], label='2th', color = 'green')
plt.plot(depth_ns_values, poly4.fit_transform(depth_ns_values.reshape(-1, 1)) [:,3], label='3th', color = 'red')
plt.plot(depth_ns_values, poly4.fit_transform(depth_ns_values.reshape(-1, 1)) [:,4], label='4th', color = 'orange')
plt.title('Polynomial Expansion vs Standardized Depth')
plt.xlabel('Standardize Depth')
plt.ylabel('h(x)')
plt.legend(); plt.xlim(-3,3)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

We can also check the arithmetic average of each polynomial basis expansion.

```python
print('The averages of each basis expansion, 0 - 4th order = ' + str(stats.describe(depth_ns_poly4)[2]) + '.')
```

    The averages of each basis expansion, 0 - 4th order = [1.         0.00536486 0.9458762  0.07336308 2.31077802].

Let's fit the linear regression model to the polynomial basis expansion.

* note the model is quite flexible to fit this complicated / nonlinear data

```python
lin_poly4 = LinearRegression()                       # instantiate new linear model
lin_poly4.fit(df_depth_ns_poly4.iloc[:,1:], Npor_ns)   # train linear model with polynomial expansion, polynomial regression
plt.subplot(111)
plt.plot(depth_ns_values, lin_poly4.predict(poly4.fit_transform(depth_ns_values.reshape(-1, 1))), label='4th order',color = 'red')
plt.plot(depth_ns, Npor_ns, 'o', label='sample data', color = 'red', alpha = 0.2, markeredgecolor = 'black')
plt.title('Standardized Porosity vs Standardized Depth')
plt.xlabel('Standardized Depth')
plt.ylabel('Standardized Porosity')
plt.xlim(-2,2); plt.ylim(-2,2)
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

```python
lin_poly4.coef_
```

    array([ 0.        ,  0.112467  ,  0.1521794 ,  0.15226789, -0.09614024])

#### Regression with Hermite Basis Expansion

We can use Hermite polynomials to reduce the correlation between the basis predictor features.

* We transformed the predicctor feature, depth, to standard normal since the Hermite poly nomials approach independence over the range of negative infinity to positive infinity under the assumption of standard normal probability density function.

```python
orders4 = [0,1,2,6,7]
depth_ns_hermite4 = scipy.special.eval_hermitenorm(orders4, depth_ns.values.reshape(-1, 1), out=None)
df_depth_ns_hermite4 = pd.DataFrame({'value':depth_ns.values,'0th': depth_ns_hermite4[:,0],'1st': depth_ns_hermite4[:,1],'2nd': depth_ns_hermite4[:,2], '5th': depth_ns_hermite4[:,3], '6th': depth_ns_hermite4[:,4]}) # make a new DataFrame from the vectors
df_depth_ns_hermite4.head()
```

      value
      0th
      1st
      2nd
      5th
      6th

      0
      -2.026808
      1.0
      -2.026808
      3.107951
      -13.948388
      -83.660777

      1
      -1.780464
      1.0
      -1.780464
      2.170053
      8.770449
      -86.669836

      2
      -1.534121
      1.0
      -1.534121
      1.353526
      20.858781
      -59.578844

      3
      -1.356312
      1.0
      -1.356312
      0.839582
      23.245520
      -31.623835

      4
      -1.213340
      1.0
      -1.213340
      0.472193
      21.929162
      -8.804757

Note: I have omitted orders that had a higher degree of correlation for our dataset.

Let's check the correlation between the Hermite predictor features. There is improvement.

```python
df_depth_ns_hermite4.iloc[:,2:].corr()
```

      1st
      2nd
      5th
      6th

      1st
      1.000000
      0.059005
      -0.109804
      0.269326

      2nd
      0.059005
      1.000000
      0.157525
      -0.047799

      5th
      -0.109804
      0.157525
      1.000000
      -0.024585

      6th
      0.269326
      -0.047799
      -0.024585
      1.000000

The pairwise linear correlation is quite low compared to the polynomial basis.

Let's visualize the bivariate relationships between our Hermite basis orders.

```python
sns.pairplot(df_depth_ns_hermite4.iloc[:,2:],vars=['1st','2nd','5th','6th'],markers='o', kind='reg',diag_kind='kde')
```

We can check the arithmetic averages of all the hermite basis expansions.

```python
print('The means of each basis expansion, 0 - 4th order = ' + str(stats.describe(depth_ns_hermite4)[2]) + '.')
```

    The means of each basis expansion, 0 - 4th order = [ 1.          0.00536486 -0.0541238   0.70118267 -1.01116309].

Let's visualize Hermite polynomials over the range of the standardized depth.

```python
plt.subplot(111)
plt.plot(depth_ns_values, scipy.special.eval_hermite(orders4, depth_ns_values.reshape(-1, 1)) [:,0], label='0th', color = 'black')
plt.plot(depth_ns_values, scipy.special.eval_hermite(orders4, depth_ns_values.reshape(-1, 1)) [:,1], label='1st', color = 'blue')
plt.plot(depth_ns_values, scipy.special.eval_hermite(orders4, depth_ns_values.reshape(-1, 1)) [:,2], label='2nd', color = 'green')
plt.plot(depth_ns_values, scipy.special.eval_hermite(orders4, depth_ns_values.reshape(-1, 1)) [:,3], label='3rd', color = 'red')
plt.plot(depth_ns_values, scipy.special.eval_hermite(orders4, depth_ns_values.reshape(-1, 1)) [:,4], label='4th', color = 'orange')
plt.title('Hermite Expansion vs Standardized Depth')
plt.xlabel('Standard Normal Depth')
plt.ylabel('h(x)')
plt.legend(); plt.ylim(-500,500); plt.xlim(-3,3)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Now let's fit our Hermite basis regression model.

```python
lin_herm4 = LinearRegression()
lin_herm4.fit(df_depth_ns_hermite4.iloc[:,1:], Npor_ns)
plt.subplot(111)
plt.plot(depth_ns_values, lin_herm4.predict(scipy.special.eval_hermitenorm(orders4, depth_ns_values.reshape(-1, 1), out=None)), label='4th order',color = 'red')
plt.plot(depth_ns, Npor_ns, 'o', label='sample data', color = 'red', alpha = 0.2, markeredgecolor = 'black')
plt.title('Hermite Polynomial: Standardized Porosity vs Standardized Depth')
plt.xlabel('Standardized Depth')
plt.ylabel('Standardized Porosity')
plt.xlim(-2,2); plt.ylim(-2,2)
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Since we have less correlation between the expanded basis features we can check out the model coefficients and interpret the unique importance of each order.

```python
lin_herm4.coef_
```

    array([ 0.        ,  0.34510167, -0.18326825,  0.00670818,  0.01108356])

#### Orthogonal Polynomials

Let's try the orthogonal polygons reimplimented in Python by Dave Moore from the poly() function in R.

* the functions below for fit and predict are directly from Dave's [blog](http://davmre.github.io/blog/python/2013/12/15/orthogonal_poly)

* note during the fit to the training data the norm2 and alpha model parameters are cacluated
* these parameters must be passed to each subsequent predict to ensure the results are consistent

```python
import numpy as np

# functions taken (without modification) from http://davmre.github.io/blog/python/2013/12/15/orthogonal_poly
# appreciation to Dave Moore for the great blog post on titled 'Orthogonal polynomial regression in Python'
# functions are Dave's reimplimentation of poly() from R

def ortho_poly_fit(x, degree = 1):
    n = degree + 1
    x = np.asarray(x).flatten()
    if(degree >= len(np.unique(x))):
            stop("'degree' must be less than number of unique points")
    xbar = np.mean(x)
    x = x - xbar
    X = np.fliplr(np.vander(x, n))
    q,r = np.linalg.qr(X)

    z = np.diag(np.diag(r))
    raw = np.dot(q, z)

    norm2 = np.sum(raw**2, axis=0)
    alpha = (np.sum((raw**2)*np.reshape(x,(-1,1)), axis=0)/norm2 + xbar)[:degree]
    Z = raw / np.sqrt(norm2)
    return Z, norm2, alpha

def ortho_poly_predict(x, alpha, norm2, degree = 1):
    x = np.asarray(x).flatten()
    n = degree + 1
    Z = np.empty((len(x), n))
    Z[:,0] = 1
    if degree > 0:
        Z[:, 1] = x - alpha[0]
    if degree > 1:
      for i in np.arange(1,degree):
          Z[:, i+1] = (x - alpha[i]) * Z[:, i] - (norm2[i] / norm2[i-1]) * Z[:, i-1]
    Z /= np.sqrt(norm2)
    return Z
```

Let's give it a try and perform orthogonal polynomial expansion of our standard normal transformed depth

```python
depth_ns_ortho4, norm2, alpha = ortho_poly_fit(depth_ns.values.reshape(-1, 1), degree = 4)
df_depth_ns_ortho4 = pd.DataFrame({'value':depth_ns.values,'0th': depth_ns_ortho4[:,0],'1st': depth_ns_ortho4[:,1],'2nd': depth_ns_ortho4[:,2], '3rd': depth_ns_ortho4[:,3], '4th': depth_ns_ortho4[:,4]}) # make a new DataFrame from the vectors
df_depth_ns_ortho4.head()
```

      value
      0th
      1st
      2nd
      3rd
      4th

      0
      -2.026808
      0.158114
      -0.330385
      0.440404
      -0.460160
      0.420374

      1
      -1.780464
      0.158114
      -0.290335
      0.313201
      -0.207862
      0.021278

      2
      -1.534121
      0.158114
      -0.250285
      0.202153
      -0.029761
      -0.172968

      3
      -1.356312
      0.158114
      -0.221377
      0.132038
      0.058235
      -0.220834

      4
      -1.213340
      0.158114
      -0.198133
      0.081765
      0.107183
      -0.219084

Let's check the correlation between the orthogonal polynomial predictor features. I'm impressed! The between basis feature order correlations are all zero!

```python
df_depth_ns_ortho4.iloc[:,2:].corr()
```

      1st
      2nd
      3rd
      4th

      1st
      1.000000e+00
      -1.387779e-16
      2.775558e-17
      -2.775558e-17

      2nd
      -1.387779e-16
      1.000000e+00
      5.551115e-17
      -2.775558e-17

      3rd
      2.775558e-17
      5.551115e-17
      1.000000e+00
      2.498002e-16

      4th
      -2.775558e-17
      -2.775558e-17
      2.498002e-16
      1.000000e+00

Let's visualize the bivariate relationships between our orthogonal polynomial basis orders.

```python
sns.pairplot(df_depth_ns_ortho4.iloc[:,2:],vars=['1st','2nd','3rd','4th'],markers='o', kind='reg',diag_kind='kde')
```

Let's visualize orthogonal polynomial basis orders over the range of the standardized depth.

```python
ortho_poly_ns_values = ortho_poly_predict(depth_ns_values.reshape(-1, 1), alpha, norm2, degree = 4)

plt.subplot(111)
plt.plot(depth_ns_values, ortho_poly_ns_values[:,0], label='0th', color = 'black')
plt.plot(depth_ns_values, ortho_poly_ns_values[:,1], label='1st', color = 'blue')
plt.plot(depth_ns_values, ortho_poly_ns_values[:,2], label='2nd', color = 'green')
plt.plot(depth_ns_values, ortho_poly_ns_values[:,3], label='3rd', color = 'red')
plt.plot(depth_ns_values, ortho_poly_ns_values[:,4], label='4th', color = 'orange')
plt.title('Orthogonal Polygon Expansion vs Standardized Depth')
plt.xlabel('Density (g/cm3)')
plt.ylabel('h(x)')
plt.legend(); plt.ylim(-.5,.5); plt.xlim(-3,3)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Finally let's fit our orthogonal polynomial basis expansion regression model.

```python
lin_ortho4 = LinearRegression()
lin_ortho4.fit(df_depth_ns_ortho4.iloc[:,1:], Npor_ns)
plt.subplot(111)

plt.plot(depth_ns_values, lin_ortho4.predict(ortho_poly_ns_values), color = 'blue', label = 'orthogonal polynomial')
plt.plot(depth_ns, Npor_ns, 'o', label='sample data', color = 'red', alpha = 0.2, markeredgecolor = 'black')
plt.title('Orthogonal Polynomial: Porosity vs Density')
plt.xlabel('Density (g/cm3)')
plt.ylabel('Porosity (%)')
plt.xlim(-2,2); plt.ylim(-2,2)
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Let's check out the model parameters for our independent orthogonal basis predictor features.

```python
lin_ortho4.coef_
```

    array([-6.73500338e-17,  2.70524700e+00, -1.44747655e+00,  1.04486290e+00,
           -9.86003333e-01])

#### Polynomial Regression in scikit-learn with Pipelines

The need to first perform basis expansion and then train the resulting (after basis transformations) linear model may seem a bit complicated.

* one solution is to use the Pipeline object from scikit-learn.

Here's some highlights on Pipelines.

###  Machine Learning Modeling Pipelines

Machine learning workflows can be complicated, with various steps:

* data preparation, feature engineering transformations

* model parameter fitting

* model hyperparameter tuning

* modeling method selection

* searching over a large combinatorial of hyperparameters

* training and testing model runs

Pipelines are a scikit-learn class that allows for the encapsilation of a seuqence of data preparation and modeling steps

* then we can treat the pipeline as an object in our much condensed workflow

The pipeline class allows us to:

* improve code readability and to keep everything straight

* avoid common workflow problems like data leakage, testing data informing model parameter training

* abstract common machine learning modeling and focus on building the best model possible

The fundamental philosophy is to treat machine learning as a combinatorial search to find the best model (AutoML)

```python
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

order=5                                                 # set the polynomial order

polyreg=make_pipeline(PolynomialFeatures(order),LinearRegression()) # make the modeling pipeline
polyreg.fit(depth_ns.values.reshape(-1, 1), Npor_ns)    # fit the model to the data
Npor_hat = polyreg.predict(depth_ns_values.reshape(-1, 1)) # predict with the modeling pipeline

plt.subplot(111)
plt.plot(depth_ns_values,Npor_hat, label='4th order',color = 'red')
plt.plot(depth_ns, Npor_ns, 'o', label='sample data', color = 'red', alpha = 0.2, markeredgecolor = 'black')
plt.title('Trained Polynomial Moder of Order = ' + str(order))
plt.xlabel('Standardized Depth')
plt.ylabel('Standardized Porosity')
plt.xlim(-2,2); plt.ylim(-2,2)
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

#### Comments

Polynomial regression is a flexible method for modeling nonlinear data and it introduces the concept of basis expansion.

* We could have done more to explore the advantages of orthogonal basis vs. nonorthogonal basis.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
On twitter I'm the @GeostatsGuy.

***

#### More on Michael Pyrcz and the Texas Center for Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
n = 100
y = np.cumsum(np.random.randn(n))

fig = plt.figure()
fig.set_size_inches(10,5)
ax = fig.add_subplot(111)
A = ax.plot(y,label='Brownian motion')
ax.legend()
```

# Brownian Motion and Stationary Processes
## Brownian Motion
$% color
% Aquamarine, black, blue, brown, cyan, darkgray, gray, green, lightgray, lime, magenta, olive, orange, pink, purple, red, teal, violet, white, yellow
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\bbspace}{\;\;\;\;\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\given}[1]{\left. #1 \right|}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\II}{\mathbb{I}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\AcA}{\mathcal{A}}
\newcommand{\FcF}{\mathcal{F}}
\newcommand{\AsA}{\mathscr{A}}
\newcommand{\FsF}{\mathscr{F}}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathcal{N}\left( #1 \right)}
\newcommand{\Exp}[1]{\mathrm{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Avar}[1]{\mathrm{Avar}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left( #1 \right)}
\newcommand{\Corr}[1]{\mathrm{Corr}\left( #1 \right)}
\newcommand{\ExpH}{\mathrm{E}}
\newcommand{\VarH}{\mathrm{Var}}
\newcommand{\AVarH}{\mathrm{Avar}}
\newcommand{\CovH}{\mathrm{Cov}}
\newcommand{\CorrH}{\mathrm{Corr}}
\newcommand{\ow}{\text{otherwise}}
\newcommand{\wp}{\text{with probability }}
\newcommand{\FSD}{\text{FSD}}
\newcommand{\SSD}{\text{SSD}}$Considering the **symmetric random walk**, which *in each time unit* is *equally likely* to *take a unit step* either to the *left* or to the *right*. That is, it's a Markov Chain with $P_{i,i+1} = P_{i,i-1} = \ffrac{1}{2}$ for $i=0,\pm 1,\dots$. Now suppose that each $\Delta t$ time unit we take a step of size $\Delta x$ either to the left or the right with equal probabilities. Let $X\P t$ denote the position at time $t$, then

$\bspace X\P t = \Delta x \P{X_1 + X_2 + \cdots + X_{\SB{t/\Delta t}}},\bspace X_i = \begin{cases}
+1,& \text{if the $i$th step of length $\Delta x$ is to the right}\\
-1,& \text{if it is to the left}
\end{cases}$

and $\SB{t/\Delta t}$ is the largest integer less than or equal to $\ffrac{t}{\Delta t}$, and the $X_i$ are assumed to be independent with $P\CB{X_i = 1} = P\CB{X_i = -1} = 0.5$. Then

$\bspace \Exp{X\P t} = \Delta x \cdot \P{0+0+\cdots+0} = 0,\bspace \Var{X\P t} = \P{\Delta x}^2 \P{1+1+\cdots+1} = \P{\Delta x}^2 \SB{\ffrac{t}{\Delta t}}$

Now let $\Delta t$ and $\Delta x$ reach $0$, and in the purpose of obtaining a limit, the path is $\Delta x = \sigma \sqrt{\Delta t}$ and thus as $t\to \infty$,

$\bspace \Exp{X\P t} = 0,\bspace \Var{X\P t} \to \sigma^2 t$

We then derive some of its properties using the **central limit theorem**:

1. $X\P t$ is normal with mean $0$ and variance $\sigma^2 t$.

2. Since the changes of value of the random walk in nonoverlapping time intervals are independent, $\CB{X\P t, t\geq 0}$ has independent increments, in that for all $t_1 < t_2 < \cdots < t_n\\[0.5em]$
$$X\P{t_n} - X\P{t_{n-1}},X\P{t_{n-1}} - X\P{t_{n-2}},\dots,X\P{t_2} - X\P{t_1},X\P{t_1}$$
are independent.

3. Since the distribution of the change in position of the random walk over any time interval depends only on the length of that interval, it would appear that $\CB{X\P t , t\geq 0}$ has stationary increments, in that the distribution of $X\P{t+s} - X\P t$ does not depend on $t$.

$Def.1$ **Brownian Motion Process**

>A stochastic process $\CB{X\P t , t\geq 0}$ is said to be a ***Brownian motion process*** if
>
>1. $X\P 0 = 0$
>2. $\CB{X\P t , t\geq 0}$ has *stationary* and *independent* increments
>3. $\forall \: t>0$, $X\P t$ is **normally distributed** with mean $0$ and variance $\sigma^2 t$
>
>And when $\sigma=1$ we call this a ***standard Brownian motion*** denoted as $B\P t = \ffrac{X\P t}{\sigma}$

In addition, $X\P t$ should be a continuous function of $t$. To prove this, we must show that

$\bspace w.p. 1,\bspace \d{\lim_{h \to 0} \P{X\P{t+h} - X\P t} = 0}$

And this seems reasonable since $X\P{t+h} - X\P t$ is a $r.v.$ with mean $0$ and variance $h$. Interestingly, although $X\P t$ will, *with probability* $1$ be a continuous function of $t$, it possesses the property of being ***nowhere differentiable***, since $\ffrac{X\P{t+h} - X\P t}{h}$ has mean $0$ and variance $1/ h$ which goes to $\infty$ as $h$ goes to $0$.

$Remark$

>The path of Brownian motion is continous (*integrable*) while NOT *derivable*, in mathematical analysis sense.
***

We now derive the joint density function of $X\P{t_1},X\P{t_2},\dots,X\P{t_n}$ for $t_1<t_2<\cdots<t_n$. The density function of $X\P t$, a normal $r.v.$ with mean $0$ and variance $t$, is

$\bspace f_t\P x = \ffrac{1}{\sqrt{2\pi t}} e^{-x^2/2t}$

Then note that

$\bspace \begin{align}
X\P{t_1}&=x_1\\
X\P{t_2}&=x_2\\
&\;\vdots\\
X\P{t_n}&=x_n
\end{align}$$\;\iff$$\begin{align}
X\P{t_1}&=x_1\\
X\P{t_2}-X\P{t_1}&=x_2-x_1\\
&\;\vdots\\
X\P{t_n} - X\P{t_{n-1}}&=x_n-x_{n-1}
\end{align}$

By the independent increment assumption, $X\P{t_1},X\P{t_2}-X\P{t_1},\dots,X\P{t_n} - X\P{t_{n-1}}$ are independent. And by the stationary increment assumption, $X\P{t_k} - X\P{t_{k-1}}$ is **normal** with mean $0$ and variance $t_k - t_{k-1}$. Hence, the joint density of $X\P{t_1},X\P{t_2},\dots,X\P{t_n}$ is given by

$\bspace\begin{align}
f\P{x_1,x_2,\dots,x_n} &= f_{t_{\void_1}}\P{x_1} \cdot f_{t_{\void_2} - t_{\void_1}}\P{x_2-x_1}\cdots f_{t_n - t_{n-1}}\P{x_{n}-x_{n-1}}\\
&= \ffrac{\exp\CB{-\ffrac{1}{2} \SB{\ffrac{x_1^2}{t_1} + \ffrac{\P{x_2-x_1}^2}{t_2-t_1} + \cdots + \ffrac{\P{x_n - x_{n-1}}^2}{t_n-t_{n-1}}}}}{\P{2\pi}^{n/2} \sqrt{t_1\P{t_2-t_1}\cdots\P{t_n-t_{n-1}}}}
\end{align}$

And with this, in principle, we can compute any desired probabilities. For instance, the conditional distribution of $X\P s$ given that $X\P t =s$ where $s<t$:

$\bspace\begin{align}
f_{s\mid t}\P{x\mid B}&= \ffrac{f_s\P x f_{t-s}\P{B-x}}{f_t\P B}\\
&= \ffrac{\exp\CB{-\ffrac{x^2}{2s} - \ffrac{\P{B-x}^2}{2\P{t-s}}}}{2\pi\sqrt{s\P{t-s}} \cdot f_t\P B}\\
&= \ffrac{\exp\CB{-\ffrac{B^2}{2\P{t-s}}}}{2\pi\sqrt{s\P{t-s}} \cdot f_t\P B}\exp\CB{-x^2\P{\ffrac{1}{2s}+\ffrac{1}{2\P{t-s}}} +\ffrac{Bx}{t-s}}\\
&= \ffrac{\exp\CB{-\ffrac{B^2}{2\P{t-s}}}}{2\pi\sqrt{s\P{t-s}} \cdot f_t\P B}\exp\CB{-\ffrac{t}{2s\P{t-s}}\P{x^2 - 2\ffrac{sB}{t}x + \ffrac{s^2B^2}{t^2} }} \exp\CB{\ffrac{t}{2s\P{t-s}}\ffrac{s^2B^2}{t^2}}\\
&= \ffrac{\exp\CB{-\ffrac{B^2}{2t}}}{2\pi\sqrt{s\P{t-s}} \cdot f_t\P B}\exp\CB{-\ffrac{t}{2s\P{t-s}}\P{x-\ffrac{sB}{t}}^2}
\end{align}$

Thus, $\Exp{X\P s \mid X\P t = B} = \ffrac{sB}{t}$ and $\Var{X\P s \mid X\P t = B} = \ffrac{s}{t}\P{t-s}$

**e.g.1**

In a two-competitor speed race, $Y\P t$ denote the amount of time, when $100t$ percent of the race has been completed, by which the racer $A$ is ahead. Suppose that we can model $\CB{Y\P t, 0\leq t\leq 1}$ as a Brownian motion process with variance parameter $\sigma^2$

$\P1$ Racer $A$ is leading by $\sigma$ seconds at the midpoint of the race, what is the probability that she is the winner?

>$\begin{align}
P\CB{Y\P 1 > 0\mid Y\P{\frac{1}{2}} = \sigma} &= P\CB{Y\P 1 - Y\P{\frac{1}{2}}>-\sigma\mid Y\P{\frac{1}{2}} = \sigma}\\
&= P\CB{Y\P 1 - Y\P{\frac{1}{2}}>-\sigma},\bspace\text{by independent increments}\\
&= P\CB{Y\P{\frac{1}{2}}>-\sigma},\bspace\text{by stationary increments}\\
&\bspace\text{since } Y\P{\frac{1}{2}}\sim\N{0,\P{\frac{\sigma}{\sqrt{2}}}^2},\\
&= P\CB{\ffrac{Y\P{1/2}}{\sigma/\sqrt{2}}>-\sqrt{2}} = \Phi\P{\sqrt{2}}
\end{align}$
>
>where $\Phi\P x = P\CB{\N{0,1}\leq x}$ is the standard normal distribution function.

$\P 2$ Given racer $A$ wins with margin of $\sigma$ seconds, what is the probability that she was ahead at the midpoint?

>To calculate $P\CB{Y\P{1/2}>0 \mid Y\P 1 = \sigma}$, we use the conclusion we just derived, that $\Exp{X\P s \mid X\P t = B} = \ffrac{sB}{t}$ and $\Var{X\P s \mid X\P t = B} = \ffrac{s}{t}\P{t-s}$. Now $X = \ffrac{Y}{\sigma} $, so the conditional distribution of $Y\P{s} = \sigma X\P s$ given that $Y\P t = C$ is normal with mean $sC/t$ and variance $\sigma^2 s\P{t-s}/t$. Hence
>
>$
P\CB{Y\P{\ffrac{1}{2}}>0 \mid Y\P 1 = \sigma} = P\CB{\N{\ffrac{\sigma}{2},\ffrac{\sigma^2}{4}}>0} = \Phi\P 1$
***

## Hitting Times, Maximum Variable, and the Gambler’s Ruin Problem
Let $T_a$ denote the *first time the Brownian Motion process hits* $a$, named, ***hitting time***. When $a>0$, we compute $P\CB{T_a \leq t}$ by conditioning on whether or not $T_a \leq t$, which leads to

$\bspace \begin{align}
P\CB{X\P t\geq a} &= P\CB{X\P t\geq a \mid T_a \leq t} \cdot P\CB{T_a \leq t} + P\CB{X\P t\geq a \mid T_a > t} \cdot P\CB{T_a > t}\\
&= P\CB{X\P t\geq a \mid T_a \leq t} \cdot P\CB{T_a \leq t}
\end{align}$

By symmetry, we have $P\CB{X\P t \geq a \mid T_a \leq t} = 0.5$ and then, based on this,

$$\begin{align}
P\CB{T_a \geq t} &= 2P\CB{X\P t \geq a}\\
&= \ffrac{2}{\sqrt{2\pi t}}\int_a^\infty \exp\CB{-\ffrac{x^2}{2t}}\;\dd x\\
&= \ffrac{2}{\sqrt{2\pi}} \int_{a/\sqrt t}^\infty \exp\CB{-\ffrac{y^2}{2}}\;\dd y, \bspace a>0
\end{align}$$

if $a<0$, then the distribution of $T_a$ is, by symmetry, the same as that of $T_{-a}$. Hence,

$\bspace P\CB{T_a \geq t} = \ffrac{2}{\sqrt{2\pi}} \int_{\abs a/\sqrt t}^\infty \exp\CB{-\ffrac{y^2}{2}}\;\dd y$

***
And the maximum value the process attains in $\SB{0,t}$, named the **maximum variable**. We obtain its distribution as follows:

$$\begin{align}
P\CB{\max_{0\leq s\leq t}X\P s\geq a} &= P\CB{T_a \leq t} \bspace\text{by continuity}\\
&= 2P\CB{X\P t \geq a}\\
&= \ffrac{2}{\sqrt{2\pi}} \int_{a/\sqrt t}^\infty \exp\CB{-\ffrac{y^2}{2}}\;\dd y
\end{align}$$

***
Now consider the probability that Brownian motion hits $A$ before $B$, where $A,B > 0$. Since Brownian motion is a limit of symmetric random walk, we start from the Gambler's ruin problem. With $N = \ffrac{A+B}{\Delta x}$ and $i = \ffrac{B}{\Delta x}$, the probability that the symmetric random walk goes up $A$ before going down $B$ is equal to $B\Delta x/\P{A+B}\Delta x = \ffrac{B}{A+B}$. Hence upon letting $\Delta x \to 0$, we see that

$\bspace P\CB{\text{up $A$ before down $B$}} = \ffrac{B}{A+B}$

## Variations on Brownian Motion
### Brownian Motion with Drift
Consider Brownian Motion $\CB{X\P t, t\geq 0}$. It's with ***drift parameter*** $\mu$ and ***variance parameter*** $\sigma^2$ if

1. $X\P 0 = 0$
2. $\CB{X\P t, t\geq 0}$ has stationary and independent increments
3. $X\P t$ is *normally* distributed with mean $\mu t$ and variance $\sigma^2 t$. Not just $\mu$ but $\mu \mathbf t$

Or equivalently, first denote $B\P t$ as the standard Brownian motion, $\CB{B\P t, t\geq 0}$, and then define $X\P t = \sigma B\P t + \mu t$

And clearly, from this representation, $B\P t$ with drift $\mu t$ is still a continous function of $t$.

### Geometric Brownian Motion
The **geomatric Brownian Motion** is defined as follows: suppose $Y\P t$ is a Brownian motion process with **drift coefficient** $\mu$ and **variance coefficient** $\sigma^2$, then $X\P t=e^{Y\P t}$ is the ***geomatric Brownian Motion***.

We now compute the expected value of the process at time $t$ given the history of the process up to time $s$ with $s<t$.

$\bspace\begin{align}
\Exp{X\P t\mid X\P u, 0\leq u\leq s} &= \Exp{e^{Y\P t}\mid Y\P u, 0\leq u\leq s}\\
&= \Exp{e^{Y\P s + Y\P t - Y\P s}\mid Y\P u, 0\leq u\leq s}\\
&= e^{Y\P s}\Exp{e^{Y\P t - Y\P s}\mid Y\P u, 0\leq u\leq s}\\
&= X\P s\Exp{e^{Y\P{t-s}}}\\
&= X\P s \cdot \exp\CB{\Exp{Y\P{t-s}} + \ffrac{1}{2}\Var{Y\P{t-s}}}\\
&= X\P s e^{\P{t-s}\P{\mu + \sigma^2/2}}
\end{align}$

$Remark$

>$\Exp{e^{Y\P t - Y\P s}\mid Y\P u, 0\leq u\leq s} = \Exp{e^{Y\P t - Y\P s}}$ is by independent increments and then $\Exp{e^{Y\P t - Y\P s}} = \Exp{e^{Y\P{t-s}}}$ is by stationary increments. And that's just the *moment generating function of a Normally Distributed* $r.v.$. And since $Y\P{t-s}\sim\N{\mu\P{t-s},\sigma^2\P{t-s}}$, we finish the calculation.

Also, it's reasonable to suppose that $X_n/X_{n-1}$, $n\geq 1$ are $i.i.d.$. Thus letting $Y_n = \dfrac{X_{n}}{X_{n-1}}$ leads to

$\bspace X_n = Y_n Y_{n-1}\cdots Y_{1}Y_0 \Rightarrow \log Y_n = \d{\sum\nolimits_{i=1}^n\log Y_i + \log X_0}$

And $Y_i$ are $i.i.d.$, then so are the $\log Y_i$. So, $\log Y_n$ is a Brownian motion with a drift after suitable normalization. Thus, $\CB{X_n}$ will also be approximately geometric Brownian motion.

## Pricing Stock Options
mostly skipped, one remain

A stochastic process $\CB{Y\P t,t\geq0}$ is said to be a ***Martingale process*** if for $s< t$,

$\bspace \Exp{Y\P t \mid Y\P u, 0 \leq u\leq s} = Y\P s$

Here're some important properties.

- if $\CB{Y\P t,t\geq0}$ is a **martingale**, then $\Exp{Y\P t} = \Exp{Y\P 0}$
- **standard Brownian motion** is a **martingale**
- $Y\P t = B^2\P t - t$ is a **martingale**
- $Y\P t = \exp\CB{c\cdot B\P t - c^2 t/2}$ is a **martingale**, where $c$ is an arbitrary constant

and the first property is especially important, at ***stopping time*** for the martingale $T$. The expected value of the stopped martingale is equal to its fixed time expecation, the ***Martingale stopping theorem***.

## The Maximum of Brownian Motion with Drift
strategically give up

## White Noise
Let $\CB{X\P t, t\geq0}$ denote a standard Brownian Motion process and let $f$ be a function having a continuous derivative in the region $\SB{a,b}$. The stochastic integral is defined as

$\bspace\d{\int_a^b f\P t \;\dd X\P t \equiv \underset{\max \P{t_{\void_i} - t_{\void_{i-1}}\:\to\: 0}}{\lim_{n\to\infty} \sum_{i=1}^n} f\P{t_{i-1}}\SB{X\P{t_i} - X\P{t_{i-1}}}}$

where $a=t_0 <t_1<\cdots t_n = b$ is a partition of the region $\SB{a,b}$. Using the identity

$\bspace\d{\sum_{i=1}^n f\P{t_{i-1}} \SB{X\P{t_{i}} - X\P{t_{i-1}}}= f\P b X\P b - f\P a X\P a - \sum_{i=1}^n X\P{t_i} \SB{f\P{t_i} - f\P{t_{i-1}}}}$

we see that

$\bspace\d{\int_a^b f\P t \;\dd X\P t = f\P b X\P b - f\P a X\P a - \int_a^b X\P t\;\dd f\P t}$

And this can be seen as the definition of $\d{\int_a^b f\P t \;\dd X\P t}$. And upon assuming the interchangeability of expecation and limit, we obtain

$\bspace \d{\Exp{\int_a^b f\P t \;\dd X\P t}} = 0$

$\bspace\begin{align}
\Var{\sum_{i=1}^n f\P{t_{i-1}} \SB{X\P{t_i} - X\P{t_{i-1}}}} &= \sum_{i=1}^n f^2\P{t_{i-1}} \Var{{X\P{t_i} - X\P{t_{i-1}}}}\\
&= \sum_{i=1}^n f^2\P{t_{i-1}} P{t_it_{i-1}}\\
\Rightarrow \Var{\int_a^b f\P t \;\dd X\P t} &= \int_a^b f^2\P t \;\dd t
\end{align}$

$Remark$

>The preceding operation, from $f$ to $\d{\int_a^b f\P t \;\dd X\P t}$, is called the ***white noise transformation***, or more loosely, $\CB{\dd X\P t, 0\leq t<\infty}$ is the ***white noise*** since it can be regarded as a time varying function $f$ travels through a white noise medium to yield the output at time $b$.

## Gaussian Processes

$Def.2$

>A stochastic process $X\P t$, $t\geq 0$ is called a ***Gaussian***, or ***Normal process***, if $X\P{t_1}$, $\dots$, $X\P{t_n}$ has a **multivariate normal distribution** for all $t_1$, $t_2$, $\dots$, $t_n$.

If $X\P t$ is already a **Brownian motion** Process, then because each of $X\P{t_1}$, $X\P{t_2}$, $\dots$, $X\P{t_n}$ can be expressed as a linear combination of the independent **normal** $r.v.$ $X\P{t_1}$, $X\P{t_2} - X\P{t_1}$, $\dots$, $X\P{t_n} - X\P{t_{n-1}}$ it follows that ***Brownian motion*** *is a* ***Gaussian process***.

And **multivariate normal distribution** can be uniquelly defined by the marginal mean values and the covariance values. It follows that standard Brownian could also be defined as a Gaussian process having $\Exp{X\P t} = 0$ and, for $s\leq t$:

$\bspace\begin{align}
\Cov{X\P s,X\P t} &= \Cov{X\P s, X\P s + X\P t - X\P s} \\
&= \Cov{X\P s,X\P s} + \Cov{X\P s, X\P t - X\P s}\\
&= \Var{X\P s} = s
\end{align}$

Note that $\Cov{X\P s, X\P t - X\P s} = 0$ because of the independent increments.

Now another special stochastic process: ***Brownian bridge***, $\CB{X\P t, 0\leq t\leq 1 \mid X\P 1 = 0}$, as it tied down both at $0$ and at $1$. Since the conditional distribution of $X\P{t_1}$, $X\P{t_2}$, $\cdots$, $X\P{t_n}$  is multivariate normal it follows that **Brownian bridge** *is a* **Gaussian process**, with $\Exp{X\P s \mid X\P 1 = 0} = 0$, for $s< 1$; and for $s<t<1$,

$\bspace \begin{align}\Cov{X\P s, X\P t \mid X\P 1 = 0} &= \Exp{X\P s X\P t \mid X\P 1 = 0} - \Exp{X\P s \mid X\P 1 = 0} \cdot \Exp{X\P t \mid X\P 1 = 0}\\
&= \Exp{\Exp{X\P s X\P t \mid X\P t, X\P 1 = 0}\mid X\P 1 = 0} - 0 - 0 \\
&= \Exp{X\P t \Exp{X\P s \mid X\P t}\mid X\P 1 = 0}\\
&= \Exp{X\P t \dfrac{s}{t}X\P t \mid X\P 1 = 0}\\
&= \dfrac{s}{t}\Exp{X^2\P t \mid X\P 1 = 0}\\
&= \dfrac{s}{t} t\P{1-t} = s\P{1-t}
\end{align}$

$Remark$

>In section [10.1](#Brownian-Motion) we've already proved that $\Exp{X\P s \mid X\P t = B} = \ffrac{s}{t}B$ and $\Var{X\P s \mid X\P t = B} = \ffrac{s}{t}\P{t-s}$

So, the **Brownian bridge** can be defined as a **Gaussian process** *with mean value $0$ and covariance function $s\P{1-t}$* for $s \leq t$.

$Proposition.1$

>If $\CB{X\P t,t\geq0}$ is **standard Brownian motion**, then $\CB{Z\P t, 0\leq t\leq 1}$ is a **Brownian bridge** process when $Z\P t = X\P t - tX\P 1$.

$Proof$

>As it is immediate that $\CB{Z\P t, t\geq 0}$ is a **Gaussian process**, all we need verify is that $\Exp{Z\P t} = 0$ and $\Cov{Z\P s,Z\P t} = s\P{1-t}$, when $s\leq t$. The former is obviously true and the latter follows from
>
>$\bspace\begin{align}
\Cov{Z\P s, Z\P t}&= \Cov{X\P s - sX\P 1, X\P t - tX\P 1}\\
&= \Cov{X\P s,X\P t} - t\cdot\Cov{X\P s , X\P 1}\\
&\bspace- s\cdot\Cov{X\P 1,X\P t}+st\cdot\Cov{X\P1,X\P1}\\
&= s - st - st + st = s\P{1-t}
\end{align}$

We also define the ***integrated Brownian motion***, as an illustration of how such a process may arice in practice. If $\CB{X\P t,t\geq0}$ is **Brownian motion**, then the **integrated Brownian motion** $\CB{Z\P t,t\geq0}$ is defined as

$\bspace \d{Z\P t = \int_0^t X\P s \;\dd s}$

thus the rate of change of $Z\P t$ follows a **Brownian motion**, $\dfrac{\dd Z\P t}{\dd t} = X\P t$ and thus

$\bspace \d{Z\P t = Z\P 0 + \int_0^t X\P s \;\dd s}$

and since **Brownian motion** is a **Gaussian motion**, **integrated Brownian motion** *is also a* **Gaussian motion**. To prove this, first recall that $W_1$, $W_2$, $\dots$, $W_n$ is said to have a multivariate normal distribution if they can be represented as

$\bspace \d{W_i = \sum\nolimits_{j=1}^m a_{ij} U_j}$, $i = 1,2,\dots,n$

where $U_j$, $j=1,2,\dots,m$ are independent normal random variables, and thus any set of partial sums of $W_1$, $W_2$, $\dots$, $W_n$ are also jointly normal.

Now $Z\P{t_1}$, $Z\P{t_2}$, $\dots$, $Z\P{t_n}$ is multivariate normal can now be shown by writing the integral form $Z\P t = \d{\int_0^t X\P s\;\dd s}$ as a limit of approximating sums. Thus it's a **Gaussian motion**.

Then, as $\CB{Z\P t, t\geq 0}$ is Gaussian it follows that its distribution is characterized by its mean value and covariance function. We now compute them under the condition that $\CB{X\P t, t\geq 0}$ is standard Brownian motion.

$\bspace\begin{align}
\Exp{Z\P t} &= \Exp{\int_0^t X\P s \; \dd s}\\
&= \int_0^t \Exp{X\P s}\;\dd s\\
&= 0
\end{align}$

and for $s\leq t$

$\bspace\begin{align}
\Cov{Z\P s,Z\P t}&= \Exp{Z\P s Z\P t} - \Exp{Z\P s}\Exp{Z\P t}\\
&= \Exp{\int_0^t X\P y \;\dd y \cdot \int_0^s X\P u \;\dd u} - 0\\
&= \int_0^s \int_0^t \Exp{X\P y X\P u}\; \dd y \dd u\\
&= \int_0^s \int_0^t \min\P{y,u}\; \dd y \dd u\\
&= \int_0^s \P{\int_0^u y\;\dd y+ \int_u^t u\;\dd y}\;\dd u = s^2\P{\dfrac{t}{2}- \dfrac{s}{6}}
\end{align}$

$Remark$

>$\Exp{X\P y X\P u} = \Cov{X\P y,X\P u} = \min\P{y,u}$ for whatever $y$ and $u$, and this shall be remembered.

***

Probability and Statistics/Applied Random Process/Chap_10_Brownian Motion and Stationary Processes.ipynb


# 第十四讲：因子分析的EM算法、主成分分析

先来回顾一下上一讲推导出的结论，对于随机变量$x=\begin{bmatrix}x_1\\x_2\end{bmatrix},\ x\sim\mathcal N(\mu,\varSigma)$，其中$\mu=\begin{bmatrix}\mu_1\\\mu_2\end{bmatrix},\ \varSigma=\begin{bmatrix}\varSigma_{11}&\varSigma_{12}\\\varSigma_{21}&\varSigma_{22}\end{bmatrix}$，则有：

* $x_1$的边缘分布$p(x_1)$为：

$$
x\sim\mathcal N\left(\mu_1,\varSigma_{11}\right)
$$

* 在$x_2$条件下$x_1$的概率$p(x_1\mid x_2)$为：

$$
\begin{align}
x_1\mid x_2&\sim\mathcal N\left(\mu_{1\mid2},\varSigma_{1\mid2}\right)\\
\mu_{1\mid2}&=\mu_1+\varSigma_{12}\varSigma_{22}^{-1}(x_2-\mu_2)\tag{1}\\
\varSigma_{1\mid2}&=\varSigma_{11}-\varSigma_{12}\varSigma_{22}^{-1}\varSigma_{21}\tag{2}
\end{align}
$$

上一讲还得到了$z,x$的联合分布为：

$$
\begin{bmatrix}z\\x\end{bmatrix}\sim\mathcal N\left(\begin{bmatrix}\vec0\\\mu\end{bmatrix},\begin{bmatrix}I&\varLambda^T\\\varLambda&\varLambda\varLambda^T+\varPsi\end{bmatrix}\right)\tag{3}
$$

所以，随机变量$x$的边缘分布为$x\sim\mathcal N\left(\mu,\varLambda\varLambda^T+\varPsi\right)$，最终得到参数的对数似然函数：

$$
\mathscr l(\mu,\varLambda,\varPsi)=\log\prod_{i=1}^m\frac{1}{\sqrt{(2\pi)^n\left\lvert\varLambda\varLambda^T+\varPsi\right\rvert}}\exp\left(-\frac{1}{2}\left(x^{(i)}-\mu\right)^T\left(\varLambda\varLambda^T+\varPsi\right)^{-1}\left(x^{(i)}-\mu\right)\right)
$$

（这里之所以选择最大化关于$p\left(x^{(i)}\right)$的似然函数，而不是关于联合分布$p\left(x^{(i)},z^{(i)}\right)$的似然函数，是因为对于给定训练集$\left\{x^{(1)},\cdots,x^{(m)}\right\}$，我们只能观测到随机变量$x$，$z$只是潜在变量。我们需要计算的是$\displaystyle\operatorname*{max}_\theta\prod_{i=1}^mp\left(x^{(i)};\theta\right)=\operatorname*{max}_\theta\prod_{i=1}^m\int_{z^{(i)}}p\left(x^{(i)},z^{(i)};\theta\right)\mathrm dz^{(i)}$。）

我们无法直接求得上面这个式子的解析解，所以转而使用EM算法逼近其最优解。

## 4. 因子分析模型的EM算法

E步骤的推导比较简单，我们需要计算$Q_i\left(z^{(i)}\right)=p\left(z^{(i)}\mid x^{(i)};\mu,\varLambda,\varPsi\right)$，将$(1),(2)$式带入$(3)$式，计算条件概率$z^{(i)}\mid x^{(i)}$：

$$
\begin{align}
z^{(i)}\mid x^{(i)};\mu,\varLambda,\varPsi&\sim\mathcal N\left(\mu_{z^{(i)}\mid x^{(i)}},\varSigma_{z^{(i)}\mid x^{(i)}}\right)\\
\mu_{z^{(i)}\mid x^{(i)}}&=\varLambda^T\left(\varLambda\varLambda^T+\varPsi\right)^{-1}\left(x^{(i)}-\mu\right)\\
\varSigma_{z^{(i)}\mid x^{(i)}}&=I-\varLambda^T\left(\varLambda\varLambda^T+\varPsi\right)^{-1}\varLambda
\end{align}
$$

将$\mu_{z^{(i)}\mid x^{(i)}},\varSigma_{z^{(i)}\mid x^{(i)}}$带入高斯分布$Q_i\left(z^{(i)}\right)$得：

$$
Q_i\left(z^{(i)}\right)=\frac{1}{\sqrt{(2\pi)^k\left\lvert\varSigma_{z^{(i)}\mid x^{(i)}}\right\rvert}}\exp\left(-\frac{1}{2}\left(z^{(i)}-\mu_{z^{(i)}\mid x^{(i)}}\right)^T\varSigma_{z^{(i)}\mid x^{(i)}}^{-1}\left(z^{(i)}-\mu_{z^{(i)}\mid x^{(i)}}\right)\right)
$$

再来看M步骤，我们需要最大化下式关于参数$\mu,\varLambda,\varPsi$的函数：

$$
\sum_{i=1}^m\int_{z^{(i)}}Q_i\left(z^{(i)}\right)\log\frac{p\left(x^{(i)},z^{(i)};\mu,\varLambda,\varPsi\right)}{Q_i\left(z^{(i)}\right)}\mathrm dz^{(i)}\tag{4}
$$

（潜在随机变量$z$服从某个高斯分布，所以这里使用积分。）

我们这里只写出关于优化$\varLambda$的推导，其他参数的推导过程类似，$(4)$式可化简为：

$$
\begin{align}
\sum_{i=1}^m\int_{z^{(i)}}Q_i\left[\log p\left(x^{(i)}\mid z^{(i)};\mu,\varLambda,\varPsi\right)+\log p\left(z^{(i)}\right)-\log Q_i\left(z^{(i)}\right)\right]\mathrm dz^{(i)}\tag{5}\\
=\sum_{i=1}^m\operatorname*{E}_{z^{(i)}\sim Q_i}\left[\log p\left(x^{(i)}\mid z^{(i)};\mu,\varLambda,\varPsi\right)+\log p\left(z^{(i)}\right)-\log Q_i\left(z^{(i)}\right)\right]\tag{6}
\end{align}
$$

这一步化简是推导中的一个重要的技巧，将积分部分看做是随机变量$z^{(i)}$在一个高斯分布$Q_i$下关于某函数的期望（$\displaystyle\operatorname*{E}_{z^{(i)}\sim Q_i}$指的服从分布$Q_i$的随机变量$z^{(i)}$的期望，后面的推导中，我们会在表达不产生歧义时省略这个下标；另外，化简计算中用到了[联合、条件及边缘概率的关系](https://en.wikipedia.org/wiki/Joint_probability_distribution#Density_function_or_mass_function)$p(x\mid z)p(z)=p(x,z)$）。继续化简，删掉式中与$\varLambda$无关的项（$z$服从高斯分布$\mathcal N(0,I)$；而$Q_i\left(z^{(i)}\right)$虽然服从高斯分布$\mathcal N\left(\mu_{z^{(i)}\mid x^{(i)}},\varSigma_{z^{(i)}\mid x^{(i)}}\right)$，但在E步骤确定了$Q_i\left(z^{(i)}\right)$之后，这个分布就固定了，其中的参数不再起变量的作用，所以这两项均与$\mu,\varSigma,\varPsi$无关），则需要最大化的目标为：

$$
\begin{align}
\sum_{i=1}^m&\mathrm E\left[\log p\left(x^{(i)}\mid z^{(i)};\mu,\varLambda,\varPsi\right)\right]\\
=&\sum_{i=1}\mathrm E\left[\log\frac{1}{\sqrt{(2\pi)^n\lvert\varPsi\rvert}}\exp\left(-\frac{1}{2}\left(x^{(i)}-\mu-\varLambda z^{(i)}\right)^T\varPsi^{-1}\left(x^{(i)}-\mu-\varLambda z^{(i)}\right)\right)\right]\\
=&\sum_{i=1}^m\mathrm E\left[-\frac{1}{2}\log\lvert\varPsi\rvert-\frac{n}{2}\log(2\pi)-\frac{1}{2}\left(x^{(i)}-\mu-\varLambda z^{(i)}\right)^T\varPsi^{-1}\left(x^{(i)}-\mu-\varLambda z^{(i)}\right)\right]
\end{align}
$$

因为$x^{(i)}\mid z^{(i)}\sim\mathcal N\left(\mu+\varLambda z,\varPsi\right)$，最大化上面这个关于$\varLambda$的函数，只有最后一项与$\varLambda$有关，于是对$\varLambda$求偏导，并使用[第二讲](chapter02.ipynb)中提到的性质：$\forall a\in\mathbb R,\mathrm{tr}a=a$、$\mathrm{tr}AB=\mathrm{tr}BA$以及$\nabla_\varLambda\mathrm{tr}ABA^TC=CAB+C^TAB$这三条性质得到（$\varLambda$是对角矩阵）：

$$
\begin{align}
\nabla_\varLambda\sum_{i=1}^m-&\mathrm E\left[\frac{1}{2}\left(x^{(i)}-\mu-\varLambda z^{(i)}\right)^T\varPsi^{-1}\left(x^{(i)}-\mu-\varLambda z^{(i)}\right)\right]\\
=\sum_{i=1}^m&\nabla_\varLambda\mathrm E\left[-\mathrm{tr}\left(\frac{1}{2}\left(z^{(i)}\right)^T\varLambda^T\Psi^{-1}\varLambda z^{(i)}\right)+\mathrm{tr}\left(\left(z^{(i)}\right)^T\varLambda^T\Psi^{-1}\left(x^{(i)}-\mu\right)\right)\right]\\
=\sum_{i=1}^m&\nabla_\varLambda\mathrm E\left[-\mathrm{tr}\frac{1}{2}\varLambda^T\varPsi^{-1}\varLambda z^{(i)}\left(z^{(i)}\right)^T+\mathrm{tr}\varLambda^T\varPsi^{-1}\left(x^{(i)}-\mu\right)\left(z^{(i)}\right)^T\right]\\
=\sum_{i=1}^m&\mathrm E\left[-\varPsi^{-1}\varLambda z^{(i)}\left(z^{(i)}\right)^T+\varPsi^{-1}\left(x^{(i)}-\mu\right)\left(z^{(i)}\right)^T\right]
\end{align}
$$

将上面的结果置为零并化简得到：

$$
\sum_{i=1}^m\varLambda\operatorname*{E}_{z^{(i)}\sim Q_i}\left[z^{(i)}\left(z^{(i)}\right)^T\right]=\sum_{i=1}^m\left(x^{(i)}-\mu\right)\operatorname*{E}_{z^{(i)}\sim Q_i}\left[\left(z^{(i)}\right)^T\right]
$$

最后，解出$\varLambda$得到：

$$
\varLambda=\left(\sum_{i=1}^m\left(x^{(i)}-\mu\right)\operatorname*{E}_{z^{(i)}\sim Q_i}\left[\left(z^{(i)}\right)^T\right]\right)\left(\sum_{i=1}^m\operatorname*{E}_{z^{(i)}\sim Q_i}\left[z^{(i)}\left(z^{(i)}\right)^T\right]\right)^{-1}\tag{7}
$$

有趣的是，这个表达式和我们从最小二乘回归得出的正规方程组$\theta^T=\left(y^TX\right)\left(X^TX\right)^{-1}$有着相似之处，可以看出$x$是关于$z$（加上噪音项）的线性函数。在E步骤中猜测$z$之后，我们尝试估计$x$与$z$之间的线性关系$\varLambda$。这个相似点到没什么，但应该注意两个算法之间的重要不同——最小二乘法会对$z$进行“最佳猜测”，我们在后面会看到这个不同。

要完成M步骤的参数$\varLambda$更新，我们需要把$(7)$式计算出来。由$Q_i$的定义可知，它是一个以$\mu_{z^{(i)}\mid x^{(i)}}$为期望，$\varSigma_{z^{(i)}\mid x^{(i)}}$为方差的高斯分布，则有：

$$
\begin{align}
\operatorname*{E}_{z^{(i)}\sim Q_i}\left[\left(z^{(i)}\right)^T\right]&=\mu_{z^{(i)}\mid x^{(i)}}^T\\
\operatorname*{E}_{z^{(i)}\sim Q_i}\left[z^{(i)}\left(z^{(i)}\right)^T\right]&=\mu_{z^{(i)}\mid x^{(i)}}\mu_{z^{(i)}\mid x^{(i)}}^T+\varSigma_{z^{(i)}\mid x^{(i)}}
\end{align}
$$

上面的第二个式子可以从协方差的定义推出，对于随机变量$Y$，$\mathrm{Cov}(Y)=\mathrm E\left[YY^T\right]-\mathrm E[Y]\mathrm E[Y]^T$，$\mathrm E\left[YY^T\right]=\mathrm E[Y]\mathrm E[Y]^T+\mathrm{Cov}(Y)$，带回$(7)$式即可得到$\varLambda$的更新规则：

$$
\varLambda=\left(\sum_{i=1}^m\left(x^{(i)}-\mu\right)\mu_{z^{(i)}\mid x^{(i)}}^T\right)\left(\sum_{i=1}^m\mu_{z^{(i)}\mid x^{(i)}}\mu_{z^{(i)}\mid x^{(i)}}^T+\varSigma_{z^{(i)}\mid x^{(i)}}\right)^{-1}\tag{8}
$$

我们一定要知道等式右边$\varSigma_{z^{(i)}\mid x^{(i)}}$的意义，这是在$x^{(i)}$条件下$z^{(i)}$的后验分布$p\left(z^{(i)}\mid x^{(i)}\right)$的协方差，而M步骤必须将这个关于$z^{(i)}$的不确定项考虑在内。一个在EM推导中常见的错误就是认为在E步骤中只需要计算潜在随机变量的期望$\mathrm E[z]$，之后用它代替M步骤优化中出现的所有“$z$的期望”就可以完成了，比如上面第二项，将$\mathrm E\left[z^{(i)}\left(z^{(i)}\right)^T\right]$直接写成$\mu_{z^{(i)}\mid x^{(i)}}\mu_{z^{(i)}\mid x^{(i)}}^T$。虽然这样确实对简单的模型（比如混合高斯模型或混合贝叶斯模型）确实有效，但这是不正确的。在我们对因子分析模型的推导中，除了$\mathrm E[z]$之外还需要$\mathrm E\left[zz^T\right]$，而且我们在上面确实看到了$\mathrm E[z]$与$\mathrm E\left[zz^T\right]$之间相差了$\varSigma_{z\mid x}$项。因此，M步骤必须将$z$变量在后验分布$p\left(z^{(i)}\mid x^{(i)}\right)$上的协方差考虑在内。

最后，我们给出M步骤中其他参数$\mu,\varPsi$的更新规则，推导这些参数并不难，首先是参数$\mu$的更新规则：

$$
\mu=\frac{1}{m}\sum_{i=1}^mx^{(i)}
$$

参数$\mu$并不像其他变量那样在更新中变化（也就是说和参数$\varLambda$不同，式子右边并不依赖$Q_i\left(z^{(i)}\right)=p\left(z^{(i)}\mid x^{(i)};\mu,\varLambda,\varPsi\right)$，也就不依赖其他参数了），这个参数只需要计算一次，并且随着算法的执行，这个参数也不需要再做更新。类似的，对角矩阵$\varPsi$的更新规则为：

$$
\varPhi=\frac{1}{m}\sum_{i=1}^mx^{(i)}\left(x^{(i)}\right)^T-x^{(i)}\mu_{z^{(i)}\mid x^{(i)}}\varLambda^T-\varLambda\mu_{z^{(i)}\mid x^{(i)}}\left(x^{(i)}\right)^T+\varLambda\left(\mu_{z^{(i)}\mid x^{(i)}}\mu_{z^{(i)}\mid x^{(i)}}^T+\varSigma_{z^{(i)}\mid x^{(i)}}\right)\varLambda^T
$$

再令$\varPsi_{ii}=\varPhi_{ii}$即可得到对角矩阵$\varPsi$（也就是让$\varPsi$只取$\varPhi$的对角线元素）。

总结整个推导过程，其中有三个地方需要注意：
* 在E步骤中我们既要计算期望，又要计算协方差；
* 在M步骤中，将积分视为某个分布的期望可以极大的简化推导；
* 注意要将协方差考虑在内。

# 第十一部分：主成分分析（Principal Components Analysis）

在上一部分的讲解中，针对“$x\in\mathbb R^n$‘近似的’分布在某个$k$维子空间中，其中$k\ll n$”情形，我们使用了因子分析模型对其建模。在讲解中我们假设将某个高斯分布的潜在变量$z^{(i)}$映射在仿射空间$\left\{\varLambda z+\mu;z\in\mathbb R^k\right\}$，然后再加上噪音项$\varPsi$后，就能够得到训练集$\left\{x^{(1)},\cdots,x^{(m)}\right\}$。因子分析是一个基于概率模型的、使用EM算法做参数估计的模型。

接下来，我们将介绍主成分分析（PCA: Principal Components Analysis），该算法同样尝试确定训练样本所“近似”分布的子空间。不同的是，PCA更加直接，而且算法只需要计算特征向量（在MATLAB中使用`eig`命令即可得到），并不需要再应用EM算法估计参数。

假设有数据集$\left\{x^{(i)};i=i,\cdots,m\right\}$，数据集代表了$m$种不同类型的汽车的属性，比如它们的最大速度、转弯半径等等，并且有$x^{(i)}\in\mathbb R^n, n\ll m$。但是有两个属性我们不知道，$x_i,x_j$，它们分别是单位为英里/小时的汽车的最大速度，和单位为公里/小时的汽车最大速度。实际上这两个属性可以说是线性相关的，只不过可能在单位转换后近似取整时造成细微的差别（比如从kph转到mph）。因此，这个数据集可以近似的看做是在$n-1$维子空间中的。那么，我们如何使用算法自动的检测或者删除像上面这种冗余特征值呢？

上面的例子可能有一种“故意设计”的感觉，那么我们来看一个实际遇到的例子：假设数据集来自关于遥控直升机操纵者的问卷调查，其中$x_1^{(i)}$代表操纵者的飞行水平、$x_2^{(i)}$代表操纵者对飞行的热爱程度。由于遥控直升机很难操作，只有那些鉴定的、衷心热爱这项运动的人才能够成为好飞行员。因此，$x_1,x_2$是强相关的。实际上，我们可能会发现数据分布在某个坐标对角线附近（下图$u_1$方向），这反映出操纵者飞行水平与热爱程度的内在因果联系，数据集中可能会存在一个背离此对角线的较小的噪音（$u_2$方向）。我们怎样才能让算法自动计算这个$u_1$方向？

接下来将介绍PCA算法，不过通常在PCA算法运行前，我们需要对数据进行预处理——对数据的期望及方差进行标准化：
1. 令$\displaystyle\mu=\frac{1}{m}\sum_{i=1}^mx^{(i)}$；
2. 将所有$x^{(i)}$替换为$x^{(i)}-\mu$；
3. 令$\displaystyle\sigma_j^2=\frac{1}{m}\sum_{i=1}^m\left(x_j^{(i)}\right)^2$；
4. 将所有$x_j^{(i)}$替换为$\displaystyle\frac{x_j^{(i)}}{\sigma_j}$。

第$(1-2)$步使数据期望归零，如果数据期望原本就是零（举个现实中的例子，比如演讲等声频信号关于时间的序列），则可以省略这两步。第$(3-4)$步会缩放样本的每个分量使其具有单位化的方差，这一步用来保证可以用相同的“尺度”对待不同的属性。举个例子，比如$x_1$是以mph表示的汽车最高时速（可能的取值为从几十到一百多），而$x_2$表示汽车座位数量（可能的取值为二到四），通过这个标准化步骤重新调整不同的属性，使它们具有可比性。如果我们具有的关于数据的先验知识表明不同的属性也在相同的尺度下，具有可比性，那么$(3-4)$步也可以省略。一个现实中的例子，比如每一个样本$x^{(i)}$表示一副灰阶图，其每个分量$x_j^{(i)}$表示第$i$幅图第$j$个像素的亮度，它们都会从$\{1,\cdots,255\}$取值，也就可以省略这个除以标准差的步骤了。

现在，标准化了数据集之后，如何计算“变量的主轴”$u$（即数据大致的分布方向）？其中一种办法是找到这样一个单位向量$u$，当我们把数据投影在$u$方向上时，投影数据的方差之和能够取最大值。直观的讲，就是数据一开始就包含某些方差/信息在内，而我们需要找到一个方向$u$，当把数据投影在由$u$指出的方向/子空间上时，能够尽可能的保留这些固有的方差/信息。

考虑下面的数据集，使用前面的标准化步骤：

现在，假设我们选择的$u$给出的方向如下图所示，图中的圆点表示原始数在该直线上的投影。

在上图中可以看到投影之后的数据依然具有很大的方差，而且远离零点。相反，如果我们按照下图选择投影直线的方向：

此时，这些投影点的方差非常小，且距离原点更近。

所以，我们希望按照前一张图中的模式让算法自动选择$u$的方向。接下来我们就正式的表达这个计算过程，对于给定向量$u$和样本向量$x$，$x$向$u$做投影，投影的长度为$x^Tu$，也就是说如果$x^{(i)}$是数据集中的一个样本点（图中的×），那么它在$u$上的投影点（图中×对应的“黑点”）到原点的距离为$x^Tu$。因此，为了最大化投影长度的方差，我们需要选择一个单位向量$u$使得下面的式子取最大值：

$$
\begin{align}
\frac{1}{m}\sum_{i=1}^m\left(\left(x^{(i)}\right)^Tu\right)^2&=\frac{1}{m}\sum_{i=1}^mu^Tx^{(i)}\left(x^{(i)}\right)^Tu\\
&=u^T\left(\frac{1}{m}\sum_{i=1}^mx^{(i)}\left(x^{(i)}\right)^T\right)u
\end{align}
$$

利用线性代数中的[关于特征向量的知识](http://nbviewer.jupyter.org/github/zlotus/notes-linear-algebra/blob/master/chapter21.ipynb)以及[关于对称矩阵的知识](http://nbviewer.jupyter.org/github/zlotus/notes-linear-algebra/blob/master/chapter26.ipynb)，我们很容易看出在$\lVert u\rVert_2=1$时最大化上面的式子将会得到对称矩阵$\displaystyle\varSigma=\frac{1}{m}\sum_{i=1}^mx^{(i)}\left(x^{(i)}\right)^T$的主特征向量，该矩阵也正好是关于数据集的协方差矩阵（假设数据的期望为零）。（如果以前没遇到过这种式子，可以尝试使用拉格朗日乘数法，在约束条件$u^Tu=1$下最大化$u^T\varSigma u$。构造拉格朗日算子$\displaystyle\mathcal L(u,\lambda)=u^T\varSigma u-\lambda\left(u^Tu-1\right)$，再对$u$求偏导$\nabla_u\mathcal L(u)=\varSigma u-\lambda u$，置为零后我们最终得到$\varSigma u=\lambda u$，也就是说$u$是$\varSigma$的特征值为$\lambda$时所对应的特征向量。）

总结一下，如果我们希望找到大致符合数据分布的一维子空间的方向$u$，那么就应该选择协方差矩阵$\varSigma$的主特征向量作为这个方向。推广到一般情况，如果我们希望将数据投影到$k$维子空间中（$k\lt n$），我们应该选择$\varSigma$的前$k$个特征向量（即特征值从大到小排列时，前$k$个特征值所对应的特征向量）作为$u_1,\cdots,u_k$。此时$u_i$成为了数据的一组新的正交基。（因为$\varSigma$是对称矩阵，则其特征向量$u_i$将相互正交，或着说总能选出相互正交的特征向量。）

现在，需要把$x^{(i)}$用这组新的基表示出来，我们只需要计算：

$$
y^{(i)}=\begin{bmatrix}u_1^Tx^{(i)}\\u_2^Tx^{(i)}\\\vdots\\u_k^Tx^{(i)}\end{bmatrix}\in\mathbb R^k
$$

与$x^{(i)}\in\mathbb R^n$相比，此时的$y^{(i)}\in\mathbb R^k$，也就是将原来的样本（$n$维）$x^{(i)}$近似表达为更低维度（$k$维）的$y^{(i)}$。因此PCA也被看做是一种降维算法，而向量$u_1,\cdots,u_k$也称作数据的前$k$个主成分。

**注意：**尽管我们只演示了$k=1$的情况，使用特征向量的性质就可以知道，在所以可能的基$u_1,\cdots,u_k$的选择中，我们选择的这组基最大化了$\displaystyle\sum_{i=1}^m\left\lVert y^{(i)}\right\rVert_2^2$。因此，这组基尽可能多的保留了原数据集中的信息。

在[问题集4](http://cs229.stanford.edu/materials/ps4.pdf)中，我们可以看到PCA的另一种推导，即选择一组能够最小化投影时产生的误差（即原数据到投影点之间的距离，图中×到对应黑点之间的距离）之和的基，这种方法同样可以得到由$k$个数据集的主成分张成的$k$维子空间。

PCA算法有很多应用，我们用几个例子来结束关于PCA的讨论。

* 首先是在数据可视化领域用来压缩数据维度，将$x^{(i)}$用维度更低的$y^{(i)}$表示，这是一个很明显的应用场景。如果我们将维数很高的数据降为$2$或$3$维，就可以将数据可视化了。比如我们将例子中的汽车相关数据降为$2$维，我们就可以画出图像（图中的一个点就代表一类车），观察哪些车与其他车类似，哪些车可以聚为一类。
* 另一种应用是做数据的预处理，在使用$x^{(i)}$做监督学习算法的输入之前先将它降维。除了减少计算量，数据降维还能降低假设类的复杂度，有助于避免过拟合（比如对于低维度输入空间的线性分类器，其VC维度较小）。
* 最后，在我们关于遥控直升机操纵者的调查中，可以看到PCA算法的降噪作用。在例子中PCA在带有噪音的关于“飞行水平”和“热爱程度”的问卷调查中，对“飞机操纵因果关系”做出了较为准确的估计。在课程中也接触到PCA关于人脸图像的应用，于是有了**特征脸（eigenfaces）**算法：输入样本$x^{(i)}\in\mathbb R^{100\times100}$是一个$10000$维向量，这是一张$100\times100$的脸部照片，向量的每一个分量都是照片中的一个像素的亮度值。使用PCA算法后，我们可以使用维度很低的$y^{(i)}$来表示$x^{(i)}$，并且希望我们找到的主成分保留了能够表达面部真实特征的有趣的、有规则的信息，并剔除了因为细微的照明变化、照片状况等因素导致的噪音。之后我们会计算降维后照片$i$与$j$之间的距离$\left\lVert y^{(i)}-y^{(j)}\right\rVert_2$。在实际中的应用结果表明，这是一个很优秀的面部匹配、检索算法。


# Trabalho Final - Análise de Dados com Python

Integrantes:

    1) Douglas Albuquerque
    2) Gaston
    3) Pedro
    4) Vitor

### Qual será seu trabalho aqui?

Basicamente, vocês devem usar todo conhecimento que adquiriram durante o curo e criar um modelo de Regressão Linear cujo RMSE seja em torno de 69000, o que significa que o erro de predição é, em média, de 69000.
Entretanto, para chegar a essa predição, uma série de análise e preparações nos dados devem ser feitas. Vocês devem fazer os todo's que aqui foram deixados a fim de chegar a esse valor numa regressão

```python
from google.colab import drive
drive.mount('/content/drive')
```

    Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

    Enter your authorization code:
    ··········
    Mounted at /content/drive

```python
# todos os pacotes que serão utilizados foram importados
# há alguns que não vimos em classe, mas parte do processo de avaliação entender como esses pacotes funcionam
# na prática, são muito similares aos que vimos em sala
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import FeatureUnion
from sklearn.preprocessing import OrdinalEncoder
from sklearn.linear_model import LinearRegression
```

```python
# leitura dos dados
housing = pd.read_csv('/content/drive/My Drive/Datasets/housing.csv')
housing.head()
```

      longitude
      latitude
      housing_median_age
      total_rooms
      total_bedrooms
      population
      households
      median_income
      median_house_value
      ocean_proximity

      0
      -122.23
      37.88
      41.0
      880.0
      129.0
      322.0
      126.0
      8.3252
      452600.0
      NEAR BAY

      1
      -122.22
      37.86
      21.0
      7099.0
      1106.0
      2401.0
      1138.0
      8.3014
      358500.0
      NEAR BAY

      2
      -122.24
      37.85
      52.0
      1467.0
      190.0
      496.0
      177.0
      7.2574
      352100.0
      NEAR BAY

      3
      -122.25
      37.85
      52.0
      1274.0
      235.0
      558.0
      219.0
      5.6431
      341300.0
      NEAR BAY

      4
      -122.25
      37.85
      52.0
      1627.0
      280.0
      565.0
      259.0
      3.8462
      342200.0
      NEAR BAY

### TODO 1: Análise e visualização de dados

    1) Use o método info e o método describe para entender como os dados se comportam
    2) percebemos que existe uma coluna cujos valores são object. Quantos valores existem para cada atributo presente nessa coluna?
    3) plote um histograma de cada coluna numérica para entender a distribuição dos dados. Use 50 bins e ajuste o tamanho das imagens para (20,15)

O que você pode observar a partir dos histogramas? Os dados podem ser inseridos num algoritmo de machine learning da maneira que estão? Dica: não!!

Tente descrever o motivo.

```python
# resposta 1

housing.info()
housing.describe()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 20640 entries, 0 to 20639
    Data columns (total 10 columns):
    longitude             20640 non-null float64
    latitude              20640 non-null float64
    housing_median_age    20640 non-null float64
    total_rooms           20640 non-null float64
    total_bedrooms        20433 non-null float64
    population            20640 non-null float64
    households            20640 non-null float64
    median_income         20640 non-null float64
    median_house_value    20640 non-null float64
    ocean_proximity       20640 non-null object
    dtypes: float64(9), object(1)
    memory usage: 1.6+ MB

      longitude
      latitude
      housing_median_age
      total_rooms
      total_bedrooms
      population
      households
      median_income
      median_house_value

      count
      20640.000000
      20640.000000
      20640.000000
      20640.000000
      20433.000000
      20640.000000
      20640.000000
      20640.000000
      20640.000000

      mean
      -119.569704
      35.631861
      28.639486
      2635.763081
      537.870553
      1425.476744
      499.539680
      3.870671
      206855.816909

      std
      2.003532
      2.135952
      12.585558
      2181.615252
      421.385070
      1132.462122
      382.329753
      1.899822
      115395.615874

      min
      -124.350000
      32.540000
      1.000000
      2.000000
      1.000000
      3.000000
      1.000000
      0.499900
      14999.000000

      25%
      -121.800000
      33.930000
      18.000000
      1447.750000
      296.000000
      787.000000
      280.000000
      2.563400
      119600.000000

      50%
      -118.490000
      34.260000
      29.000000
      2127.000000
      435.000000
      1166.000000
      409.000000
      3.534800
      179700.000000

      75%
      -118.010000
      37.710000
      37.000000
      3148.000000
      647.000000
      1725.000000
      605.000000
      4.743250
      264725.000000

      max
      -114.310000
      41.950000
      52.000000
      39320.000000
      6445.000000
      35682.000000
      6082.000000
      15.000100
      500001.000000

```python
# resposta 2

housing['ocean_proximity'].value_counts()

```

    <1H OCEAN     9136
    INLAND        6551
    NEAR OCEAN    2658
    NEAR BAY      2290
    ISLAND           5
    Name: ocean_proximity, dtype: int64

```python
# resposta 3

housing.hist(bins=50, color='blue')
plt.figure(figsize=(20,15))
```

### TODO 2: Criando conjunto de treino e teste
    4) utilizando train_test_split, da Scikit-Learn, separe o conjunto em treino e teste. utilize train_set e test_set como nome de variáveis, separe 20% para conjunto de teste e utilize 42 como random_state

```python
# resposta 4

#feature_col = housing.iloc[:,0:6]
#meta_col =  housing.iloc[:,7]
feature_col = housing.loc[:,['longitude','latitude','housing_median_age','total_rooms','total_bedrooms','population','households','median_income']]
meta_col = housing.loc[:,['median_house_value']]

X = feature_col.values
y = meta_col.values

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8,test_size=0.2, random_state=42)

```

### TODO 3: Correlação e visualização

    5) Quais as variáveis que mais possuem correlação com o atributo median_house_value?
    6) Utilize scatter_matrix, do pandas para visualizar a correlação do atributo median_house_value com os atributos que possuem maior correlação com ele. Ajuste o tamanho das imagens para (12, 8)

```python
# resposta 5

import seaborn as sns
plt.figure(figsize=(12,10))
cor = housing.corr()
sns.heatmap(cor, annot=True, cmap=plt.cm.Reds)
plt.show()
```

```python
# resposta 6

pd.plotting.scatter_matrix(housing, diagonal='kde')
```

### TODO 4: Limpeza de dados

Ao usar o método info(), descobrimos que existem alguns dados faltantes no nosso algoritmo. A partir daqui, a variável housing será o conteúdo de train_set, excluindo-se a coluna meta (median_house_value).  Atribua para a variável housing_labels uma cópia de train_set só com informações da coluna meta

    7) use simpleImputer para tratar os valores faltantes. Escolha a estratégia que achar mais conveniente

```python
# resposta 7

#Quantidade de linhas em branco no dataset
vazios = housing.isnull().sum(axis = 0)

print('Colunas que possuem linhas vazias: ',vazios)

```

    Colunas que possuem linhas vazias:  longitude               0
    latitude                0
    housing_median_age      0
    total_rooms             0
    total_bedrooms        207
    population              0
    households              0
    median_income           0
    median_house_value      0
    ocean_proximity         0
    dtype: int64

```python
# resposta 7 (continuação)

housing_labels = X

imputer = SimpleImputer(missing_values=np.nan, strategy='median')
#imputer = imputer.fit(housing_labels[3])

housing_labels
```

    array([[-1.2223e+02,  3.7880e+01,  4.1000e+01, ...,  3.2200e+02,
             1.2600e+02,  8.3252e+00],
           [-1.2222e+02,  3.7860e+01,  2.1000e+01, ...,  2.4010e+03,
             1.1380e+03,  8.3014e+00],
           [-1.2224e+02,  3.7850e+01,  5.2000e+01, ...,  4.9600e+02,
             1.7700e+02,  7.2574e+00],
           ...,
           [-1.2122e+02,  3.9430e+01,  1.7000e+01, ...,  1.0070e+03,
             4.3300e+02,  1.7000e+00],
           [-1.2132e+02,  3.9430e+01,  1.8000e+01, ...,  7.4100e+02,
             3.4900e+02,  1.8672e+00],
           [-1.2124e+02,  3.9370e+01,  1.6000e+01, ...,  1.3870e+03,
             5.3000e+02,  2.3886e+00]])

```python
housing_labels.describe()
```

### TODO 4: Manipulação de Atributos categóricos
    7) Utilizando OrdinalEncoder, manipule os valores categóricos para preparar os dados corretamente para algoritmos de machine learning

```python
# resposta 7
```

### TODO 5: Feature Normalization

Com algumas poucas exceções, algoritmos de machine learning não performam bem quando os atributos numéricos possuem escalas diferentes. Este é o caso de nosso dataset, em que o atributo total_rooms possui um range de 6 a 39320 enquanto median_income varia de 0.5 a 15.

Existem duas abordagens conhecidas para trazer os atributos para uma mesma escala: min-max scaling e standardization.

min-max scaling (normalization) consiste em subtratir o valor minimo e dividir pela subtração de max por min:

\begin{align}
x = \frac{x - min}{max - min}
\end{align}

A Scikit-Learn provê o método MinMaxScaler que realiza essa operação.

Já standardization consiste em subtrair a média(valores sempre terao média zero) e então dividir pela variância de modo que a distribuição resultante possua variância unitária.

\begin{align}
x = \frac{x - \mu}{\sigma}
\end{align}

Diferentemente de normalization, standardization não leva os números a um intervalo específico, o que pode ser um problema para algoritmos específicos (por exemplo, redes neurais geralmente esperam um valor de entrada entre o intervalo 0-1). Entretanto, standdardization é muito menos sensível à outiliers. Scikit-learn provêo método StandardScaler para standardization.

    8) Adeque os valores de cada coluna numérica para que eles fiquem normalizados. Use o método que achar mais conveniente

```python
# resposta 8
```

### TODO 6: Pipeline de Transformação
    9) crie um pipeline de transformação para tratar os atributos numéricos, outro para tratar os atributos categóricos e um para juntar os dois e preparar todos os dados para entrada num algoritmo de machine learning. Caso queira, pode usar a classe DataFrameSelector vista na aula 4 (recomendável). A variável de saída desse algoritmo deverá se chamar housing_prepared

```python
# resposta 9
```

### TODO 7: Regressão Linear

    10) Ajuste uma regressão linear aos dados. Utilize a versão da Scikit-Learn

```python
# resposta 10
```

```python
# Apenas para testar seu resultado
some_data = housing.iloc[:5] # dados para predizer
some_labels = housing_labels.iloc[:5] # rótulos dos dados acima
some_data_prepared = full_pipeline.transform(some_data) # transformação dos dados (considerando que seu pipeline tenha esse nome)
print("Predictions:\t", lin_reg.predict(some_data_prepared)) # predizendo
print("Labels:\t\t", list(some_labels)) # valores corretos
```

### TODO 8: RMSE
    11) Utilizando Scikit-Learn, extraia o RMSE das predições a partir da variável housing_prepared.

```python
# resposta 11
```


# oommf Slonczewski code

## betaeps and betaepsp

```
import sympy
sympy.init_printing(use_latex=True)
```

```
alpha, gamma, Lambda, P, beta, m, p, J, Ms, eps, epsp = sympy.symbols("alpha, gamma, Lambda P beta m p J Ms epsilon \epsilon'")
```

```
Lambda_fixed_init = Lambda  # Lambda
Lambda_free_init = Lambda_fixed_init  # Lambda
```

```
P_fixed_init = P  # P
P_free_init = P_fixed_init  # P
```

```
beta_q_plus = Lambda_fixed_init  # Lambda
beta_q_minus = Lambda_free_init  # Lambda
```

```
fixed_temp = beta_q_plus  # Lambda
free_temp = beta_q_minus  # Lambda
```

```
fixed_temp = fixed_temp * fixed_temp  # Lambda^2
free_temp = free_temp * free_temp  # Lambda^2
```

```
fixed_factor_plus = sympy.sqrt(fixed_temp + 1.0)  # sqrt(Lambda^2 + 1)
free_factor_plus = sympy.sqrt(free_temp + 1.0)  # sqrt(Lambda^2 + 1)
plus_ratio = free_factor_plus / fixed_factor_plus  # 1
print "plus_ratio = {}".format(plus_ratio)
```

    plus_ratio = 1

```
fixed_factor_minus = sympy.sqrt(fixed_temp - 1.0)  # sqrt(Lambda^2 - 1)
free_factor_minus = sympy.sqrt(free_temp - 1.0)  # sqrt(Lambda^2 - 1)
minus_ratio = fixed_factor_minus / free_factor_minus  # 1
print "minus_ratio = {}".format(minus_ratio)
```

    minus_ratio = 1

```
B = fixed_factor_minus * free_factor_minus  # Lambda^2 - 1
A = fixed_factor_plus * free_factor_plus  # Lambda^2 + 1
A, B
```

```
beta_q_plus = beta * 0.5 * fixed_temp * plus_ratio  # Lambda^2 * beta / 2
beta_q_minus = beta * 0.5 * free_temp * minus_ratio  # Lambda^2 * beta / 2
beta_q_plus, beta_q_minus
```

```
beta_q_plus = beta_q_plus * P_fixed_init  # P * Lambda^2 * beta / 2
beta_q_minus = beta_q_minus * P_free_init  # P * Lambda^2 * beta / 2
beta_q_plus, beta_q_minus
```

```
fixed_temp = beta_q_plus  # P * Lambda^2 * beta / 2
free_temp = beta_q_minus  # P * Lambda^2 * beta / 2

beta_q_plus = fixed_temp + free_temp  # P * Lambda^2 * beta
beta_q_minus = fixed_temp - free_temp  # 0 ... this simplies things!!!
beta_q_plus, beta_q_minus
```

```
Bdot = B
Bdot = Bdot * m * p
Bdot
```

```
Aplus = A + Bdot  # (Lambda^2 + 1) + (Lambda^2 - 1)(m * p)
Aminus = A - Bdot  # (Lambda^2 + 1) - (Lambda^2 - 1)(m * p)
Aplus, Aminus
```

```
betaeps = beta_q_plus/Aplus + beta_q_minus/Aminus
betaeps
```

```
betaepsp = beta * epsp
betaepsp
```

```
betaeps = betaeps
betaepsp = betaepsp
betaeps, betaepsp
```

## LLG + Slonczewski

```
coef1 = alpha * betaeps - betaepsp
coef1
```

```
coef2 = alpha * betaepsp + betaeps
coef2
```

```
mxH = sympy.symbols("(m\\times{}H)")
scratchA = mxH
scratchA = alpha * mxH
scratchA
```

```
mxp = sympy.symbols("(m\\times{}p)")
scratchB = mxp * coef2
scratchA = scratchA + scratchB
scratchA
```

Things get messy because of cross products, which I will not represent correctly.

```
scratchA = scratchA * m
scratchA
```

```
mxp = mxp * coef1
mxp
```

```
scratchA = scratchA + mxp
scratchA
```

```
scratchA = scratchA - mxH
scratchA
```

```
scratchA = gamma * scratchA
scratchA
```

```

```


```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
```

```python

```

## Aula dia 24/01

```python
# Inicios
P = 24E3 #Cargas
n = 3    #Qtd de parafusos

# Encontrar o centroide
df = pd.DataFrame()
df['Xi'] = [-150, 150, 0]
df['Yi'] = [0, 0, -300]

df.sum()/3
```

    Xi      0.0
    Yi   -100.0
    dtype: float64

```python
# Encontrando o vetor posição r
r1 = np.sqrt(150**2 + 100**2)
r2 = r1
r3 = 300-100

# Momento
M = 150E-3 * P

```

```python
F1i = P/n
F1i
```

    8000.0

```python
5/1.414
```

    3.5360678925035365

## Aula 25/01 - Projeto de soldas

```python
from sympy import *
from sympy.abc import *
import numpy as np

Aw = pi*d
Sw = (pi*d**2)/(4)
Jw = (pi*d**3)/4

```

```python

a = 0.175
r = 0.120

#Cortante
fs = P/Aw

#Momento flexão
fb = (P*a)/(Sw)

#Torção
ft = ((P*r)*(d/2))/Jw
```

```python
Fr = sqrt(fs**2 + fb**2 + ft**2)
Fr
```

$\displaystyle \sqrt{\frac{P^{2}}{d^{2} \pi^{2}} + \frac{0.5476 P^{2}}{d^{4} \pi^{2}}}$

```python
sigma = 124E6
t = Fr/sigma
#a = Eq(Fr, 0.005*np.cos(np.radians(45)))
t
```

$\displaystyle 8.06451612903226 \cdot 10^{-9} \sqrt{\frac{P^{2}}{d^{2} \pi^{2}} + \frac{0.5476 P^{2}}{d^{4} \pi^{2}}}$

```python
exp = Eq(t, 0.005*np.cos(np.radians(45)))
num = exp.subs({d:0.025, pi:np.pi})
solve(num) #Newtons de força
```

    [-1162.59155406151, 1162.59155406151]

```python

```

## Teste 5 -

```python
from sympy import *
from sympy.abc import *
import numpy as np
import matplotlib.pyplot as plt

init_printing()

import warnings
warnings.filterwarnings("ignore")

sa, sm, sf_, sut, nf, ktf, ktt = symbols("\sigma'_a \sigma'_m S_f S_{ut}, N_f k_{tf} k_{tt}")

#Função pra calcular sf'
def Sf_l(kc, kd, sf):
    ka = 1 #flexão equivalente
    kb = 1.189*((D*1000)**-0.097)
    #kb = D*1.189*((1000)**-0.097)
    return(ka*kb*kc*kd*(0.5*sf))

#Critério de falha
falha = Eq(sa/sf_ + sm/sut, 1/nf)
falha
```

```python
ka = 1
kb = 1.189*((d*1000)**-0.097) # expressão é em mm por isso *1000 -> transformar m (d) to mm p/ expressão
kc = 0.75
kd = 0.814

r = ka*kb*kc*kd*372.5
r = r.subs({d:1})
r

num = falha.subs({sa:35051.79, sf_:1000*a*d**3, sm:35051.79, sut:1000*108*d**3, nf:2})
num
```

```python
solve(num)
```

```python
ka = 1
kb = 1.189*((d*1000)**-0.097) # expressão é em mm por isso *1000 -> transformar m (d) to mm p/ expressão
kc = 0.75
kd = 0.814

r = ka*kb*kc*kd*372.5
r = r.subs({d:1})
r

num = falha.subs({sa:1619.46, sf_:1e6*r*d**3, sm:1619.46, sut:1e6*745*d**3, nf:2})
num

solve(num)
```

```python

```