# W261  MIDS Machine Learning at Scale
## End of Term exam

Spring, 2018
__Student Name__: Leslie TEO
__Student ID:__   303218617

__Email:__   lteo01@berkeley.edu | lteo@ischool.berkeley.edu

# Instructions for exam

* Please keep all your work and responses in ONE (1) notebook only (and submit to your github repository in a directory named FINAL)

* Please submit your solutions and notebook via ISVC. Go to: Coursework -> Assignments & Grading -> End of Term Exam

* **Question types:**
*    Knowledge test Programmatic/doodle (take photos; embed the photos in your notebook) and put them on GitHub
*    All programmatic questions can be run locally on your laptop (using Apache Spark only)

* This is an open book exam meaning you can consult webpages and textbooks, class notes, slides etc. but you can not discuss with each other or any other person/group until after the course finishes.

* If there is any collusion, then this will result in a zero grade and will be grounds for dismissal from the entire program for those involved. Please complete this exam by yourself within the time limit.

7. For markdown help in iPython Notebooks please see:
https://sourceforge.net/p/ipython/discussion/markdown_syntax

# Exam helper functions:

### Questions requiring a SINGLE numeric response

```python
import numpy as np
# for two decimal places;
# change this to the desired precision as needed
ans=3.145  #replace with your answer
print("%.2f"%np.round(ans, 2))
#prints 3.14
#if this was your response then paste 3.14 (no spaces before or after; no extra characters; just 4 character: 3, ., 1, 14)
```

    3.14

### Questions requiring a MULTIPLE numeric responses

```python
import numpy as np
# for two decimal places;
# change this to the desired precision as needed
ans1=1.145  #replace with your answer
ans2=2.44333
ans3=3.444443
print("%.2f,%.2f,%.2f" %(np.round(ans1, 2), np.round(ans2, 2), np.round(ans3, 2)))
#prints 1.14,2.44,3.44
#if this was your response then paste this string 1.14,2.44,3.44 (no spaces before or after; no extra characters)
```

    1.14,2.44,3.44

### Import libraries, set up spark

```python
import numpy as np
import re
import matplotlib.pyplot as plt
%matplotlib inline

plt.style.use('ggplot')

%reload_ext autoreload
%autoreload 2
```

```python
from pyspark.sql import SparkSession
app_name = "Notebook_FINAL"
master = "local[*]"
spark = SparkSession\
        .builder\
        .appName(app_name)\
        .master(master)\
        .getOrCreate()
sc = spark.sparkContext
```

```python
spark
```

SparkSession - in-memory

SparkContext

Spark UI

Version

v2.3.0

Master

local[*]

AppName

Notebook_FINAL

```python
sc
```

SparkContext

Spark UI

Version

v2.3.0

Master

local[*]

AppName

Notebook_FINAL

# Exam questions begins here

##  ET:1
Calculate the signed normalizaed perpendicular distance from hyperplane $4x + 3y = 2$ to the point $(1, 3)$
Please provide a response  within two decimal digits of precision in your notebook the submission form.

```python
import numpy as np
import os

import matplotlib.pyplot as plt
%matplotlib inline
x = np.arange(-2, 5)
plt.plot(x, (2. - 4. * x) / 3.)
plt.gca().set_aspect('equal', adjustable='box')
plt.scatter([1],[3], c="r", s=40)
plt.xlim([-2, 4])
plt.ylim([-2, 4])
plt.grid(1)
plt.show()
```

```python
plane = [4,3,-2]
point = [1,3]
normal_vector = plane[:-1]
normal_vector_magnitude = np.sqrt(np.dot(normal_vector, normal_vector))
normal_unit_vector = normal_vector / normal_vector_magnitude

#print normal_vector
#print normal_vector_magnitude
#print normal_unit_vector

perpendicular_distance = (np.dot(plane, point + [1]))/ normal_vector_magnitude
print("%.2f"%np.round(perpendicular_distance, 2))

```

    2.20

**Answer: 2.20 **

##  ET:2
Consider a three-class classification problem with the following separating hyperplanes:

$$
\begin{align}
4x + 3y - 9 = 0 & & 1^{\text{st}} \text{ class}\\
-4x + 3y - 11 = 0 & & 2^{\text{nd}} \text{ class}\\
-x - 3y - 1 = 0 & & 3^{\text{rd}} \text{ class}
\end{align}
$$

Calculate the score (signed unnormalized perpendicular distance) for each class and the point (0, 1).

As a response provide all class scores within two decimal digits precision.
Provide your response in your notebook here and in the submission form.
Use the above provided helper functions to format your answers.

```python
import numpy as np
import os
import matplotlib.pyplot as plt

x = np.arange(-3, 5)
plt.plot(x, (9. - 4. * x) / 3., label="1st")
plt.plot(x, (11. + 4. * x) / 3., label="2nd")
plt.plot(x, (-1. - 1. * x) / 3., label="3rd")
plt.gca().set_aspect('equal', adjustable='box')
plt.scatter([0], [1], c="k", s=40)
plt.xlim([-3, 4])
plt.ylim([-3, 4])
plt.legend()
plt.grid(1)
plt.show()
```

```python
# 1st Class
plane = [4,3,-9]
point = [0,1]
normal_vector = plane[:-1]
normal_vector_magnitude = np.sqrt(np.dot(normal_vector, normal_vector))

perpendicular_distance = np.dot(plane, point + [1])/normal_vector_magnitude
print("%.2f"%np.round(perpendicular_distance, 2))
```

    -1.20

```python
# 2nd Class
plane = [-4,3,-11]
point = [0,1]
normal_vector = plane[:-1]
normal_vector_magnitude = np.sqrt(np.dot(normal_vector, normal_vector))

perpendicular_distance = np.dot(plane, point + [1])/normal_vector_magnitude
print("%.2f"%np.round(perpendicular_distance, 2))
```

    -1.60

```python
# 3rd Class
plane = [-1,-3,-1]
point = [0,1]
normal_vector = plane[:-1]
normal_vector_magnitude = np.sqrt(np.dot(normal_vector, normal_vector))

perpendicular_distance = np.dot(plane, point + [1])/normal_vector_magnitude
print("%.2f"%np.round(perpendicular_distance, 2))
```

    -1.26

**Answer**: ** 1st class: -1.20; 2nd class: -1.60; 3rd class: -1.26 **

##  ET:3
Take the data from question ET2  above and calculate the class conditional probabilities for the point (0, 1) with respect to each class in this multinomial logistic regression model.

As a response provide all class conditional probabilities within **five** decimal digits precision.
Use the helper functions above to format your answer. Make sure to adjust it to produce 5 decimal precision digits.

```python
# Scores for (0,1)
score_1 = -1.20
score_2 = -1.60
score_3 = -1.26

# Total unormalized scores
total = np.exp(score_1) + np.exp(score_2) + np.exp(score_3)

print 'Conditional probability of class 1: %.5f' %np.round(np.exp(score_1)/total,5)
print 'Conditional probability of class 2: %.5f' %np.round(np.exp(score_2)/total,5)
print 'Conditional probability of class 3: %.5f' %np.round(np.exp(score_3)/total,5)
```

    Conditional probability of class 1: 0.38284
    Conditional probability of class 2: 0.25662
    Conditional probability of class 3: 0.36054

**Answer**: **Class 1: 0.38284; Class 2: 0.25662; Class 3: 0.36054 **

## ET:4
Take the data from question ET3 above and predict the right class using the above multinomial logistic regression model.

Select the correct response from the following.

(a) Class 1
(b) Class 2
(c) Class 3
(d) all classes are equally likely

**Answer**: **(a) Class 1 **

##  ET:5
In the following (and also referring to HW12: http://nbviewer.jupyter.org/urls/dl.dropbox.com/s/1wb2rdqbet54y1h/MIDS-MLS-Project-Criteo-CTR.ipynb) we have hashed the three sample points using numBuckets=4 and numBuckets=100. Complete the  statement below about these hashed features summarized in the following table using each answer once.

|Name|Raw Features|4 Buckets|100 Buckets|
|--|--|--|--|
|sampleOne|[(0, 'mouse'), (1, 'black')]|{2: 1.0, 3: 1.0}|{14: 1.0, 31: 1.0}|
|sampleTwo|[(0, 'cat'), (1, 'tabby'), (2, 'mouse')]|{0: 2.0, 2: 1.0}|{40: 1.0, 16: 1.0, 62: 1.0}|
|sampleThree|[(0, 'bear'), (1, 'black'), (2, 'salmon')|{0: 1.0, 1: 1.0, 2: 1.0}|{72: 1.0, 5: 1.0, 14: 1.0}|

With 4 buckets, sampleTwo and sampleThree both contain index 0 due to __________.

(a) A hash collision
(b) Underlying properties of the data
(c) The fact that we use 4 buckets
(d) none of the above

**Answer**: **(a)**

##  ET:6
When applying numerical machine learning approaches (and for non-numerical approaches if required) to big data problems which of the following steps are could be used during modeling and are recommended:

Select one:
(a) Convert categorical features to numerical features via one-hot-encoding and store in a dense representation
(b) Transform  categorical features using hashing regardless of how many unique categorical values exist in training and test data
(c) Use matrix factorization to remap your input vectors to latent concepts
(d) none of the above

**Answer**: **(b) is the best answer.** [(a) is true if it is stored in a sparse representation, for (b) "regardless" might be too strong depending on number and categories reduced to, (c) does not make sense.]

##  ET:7
When dealing with numercial data which of the following are ways to deal with missing data:

Select one:
(a) Delete records that have missing input values
(b) Standardize the data and set all missing values to 1 (one)
(c) Use K-nearest neighbours based on the test set to fill in missing values in the training set
(d) none of the above

**Answer**: **(a)** [Although not the best, (b) may not work since 1 means something, (c) works but we are contaminating test and training data.]

##  ET:8
In the Criteo project, we're trying to predict what:

Select one:
(a) Revenue from click events
(b) Click-through vs not click event
(c) Probability of a purchase
(d) none of the above

**Answer**: **(b)**

##  ET:9
Which of the following are true about the purpose of a loss function?

Select one or more:
(a) It’s a way to penalize a model for incorrect predictions
(b) It precisely defines the optimization problem to be solved for a particular learning model
(c) Loss functions can be used for modeling both classification and regression problems
(d) none of the above

**Answer**: **(a), (b), (c) **

##  ET:10
Many standard machine learning methods can be formulated as a convex optimization problem, i.e. the task of finding a minimizer of a convex function _f_ that depends on a variable vector __w__ (called weights in the code), which has _d_ entries. Formally, we can write this as the optimization problem minw∈ℝdf(w), where the objective function is of the form  (JGS)

\begin{equation*}
minimize   \left(f(w):= λR(w) + \sum_{k=1}^n \left(w;w_i,y_i \right) \right)
\end{equation*}

Here the vectors xi∈ℝd are the training data examples, for 1≤i≤n, and yi∈ℝ are their corresponding labels, which we want to predict. We call the method linear if _L(w;x,y)_ can be expressed as a function of _ __wT__x _ and y. Several of spark.mllib’s classification and regression algorithms fall into this category.

The objective function __f__ has two parts: the regularizer that controls the complexity of the model, and the loss that measures the error of the model on the training data. The loss function _L(w;.)_ is typically a convex function in __w__. The fixed regularization parameter λ≥0λ≥0 (regParam in the code) defines the trade-off between the two goals of minimizing the loss (i.e., training error) and minimizing model complexity (i.e., to avoid overfitting).

When implementing Logistic (or linear) Regression with Regularization in Spark which of the following apply when using the above cost functions (mulitple options may apply):

(I)   When lambda equals one, it provides the same result as standard logistic/linear regression
(II)  One only needs to modify the standard logistic (linear) regression (i.e., with no regularization term) by adding some code after the map-reduce (loss) gradient steps
(III) When lambda equals zero, it provides the same result as standard logistic (linear) regression
(IV)  None of the above

Select the most correct from the following:

* (a) I, II, III
* (b) II, III
* (c) III
* (d) IV

**Answer**: **(b)**

##  ET:11
In the context of ecommerce you have just deployed a new conversion rate prediction model to production. This model (aka treatment model) will challenge the control nodel (i.e., the current model) in AB Test manner to see if it can be produce better revenue. Here is the data that was taken from this live AB Test.

```
CONTROL MODEL (our new CTR model)
Impression ID   Revenue
1                $0.50
2                $0.50
3                $3.00
......
20000            $3.00
20001            $3.00
20002            $3.00
20003            $3.00
......
50,001           $3.00
.....
100,000          $4.00
```

All other impressions in this 100,000 sample resulted in zero transactions and therefore zero revenue.

```
TREATMENT MODEL (our new CTR model)
Impression ID   Revenue
1                $1.50
2                $0.50
3                $0.00
......
50,001           $3.00
.....
100,000          $4.00
```
All other impressions in this 100,000 sample resulted in zero transactions and therefore zero revenue.

P-values are a common way to determine the statistical significance of a test. The smaller it is, the more confident you can be that the test results are due to something other than random chance.
A common p-value of .05 is a 5% significance level. Similarly, a p-value of .01 is a 1% significance level. A p-value of .20 is a 20% significance level. For this problem set the p-value to 0.01

Which of the following are true:

(a) Based on revenue there is no statistical significant difference between the Control and the Treatment at p-value of 0.05 for a one-sided t-test
(b) Based on transaction rates (tranactions that generated revenue versus not) there is no statistical significant difference between the Control and the Treatment at p-value of 0.05 for a one-sided t-test
(c) AB testing using differences in revenue for this problem is a  useful means of determining if the Treatment conversion rate prediction model is better than the control model.
(d) none of the above

```python
 # Note discrepancy between this code and the exam
import numpy as np
from scipy.sparse import coo_matrix
import scipy.stats

control=[.5, .5, 3, 3, 3, 3, 3, 3, 4]
control.extend([0]*(100000-len(control)))
control=np.asarray(control)

treatment=[ 1.5, .5, 0, 0, 0, 0, 0, 3, 4]
treatment.extend([0]*(100000-len(treatment)))
treatment=np.asarray(treatment)

#nb 2-sided test
scipy.stats.ttest_ind(control, treatment), scipy.stats.ttest_ind(control.astype(bool), treatment.astype(bool))
```

    (Ttest_indResult(statistic=1.4142505069117979, pvalue=0.1572899198561911),
     Ttest_indResult(statistic=1.3867952958906298, pvalue=0.1655057375790788))

**Answer**: **(a) ** [Scipy gives us a 2-tailed test so we have to divide it's output by 2. In both cases, The p-value is still greater than 0.05. NOTE: In exame we only have one choice. Here it implies we can have more than one.]

##  ET:12
Given this graph expressed in the form of an adjacency list,

```
Node  adjacentNode:weightAssociatedWithEdge
N1    N6:10,  N2:2
N2    N3:1
N3    N4:1
N4    N5:1
N5    N6:1
N6    N7:1
N7    N8:1
N8    N9:1
```

Using the parallel breadth-first search algorithm for determining the shortest path from a single source, how many iterations are required to discover the shortest distances to all nodes from Node 1

A 7
B 8
C 13
D None of the above

**Answer**: **(b) ** One iteration for each node to node 1 => 9-1 iterations

##  ET:13
Many standard machine learning methods can be formulated as a convex optimization problem, i.e. the task of finding a minimizer of a convex function f that depends on a variable vector w (called weights in the code), which has d entries. Formally, we can write this as the optimization problem where the objective function is of the form

$$
f(w):= λR(w) + \frac{1}{n}\sum_{i=1}^n L\left(w;w_i,y_i \right)
$$

Here the vectors $x_i \in ℝ_d$ are the training data examples, for $1≤i≤n$, and $y_i \in ℝ$ are their corresponding labels, which we want to predict.

We call the method linear if $L(w;x,y)$ can be expressed as a function of $wTx$ and $y$. (where $wT$ denotes the transpose of the weight vector or matrix w )

By default, linear SVMs are trained with an L2 regularization with a hinge loss that is defined as follows:  $max\{0,1−ywTx\},y \in \{−1,+1\}$

Given the a softSVM as defined above select the most correct combination of statements from the following:

(I) When λ is super small (e.g., 0.000001), then the above Langrangian will yield a Hard SVM approximately
(II) In the context of support vector machines, linear kernels can be readily parallelized in map reduce frameworks such as Spark
(III) Sequential learning via algorithms such as perceptron can take advantage of map-reduce frameworks and yield   exactly the same results as a single core implementation with significant reductions in training time
(IV) When λ is 1.0, then the above Langrangian will yield a Soft SVM

Select one:

(a) I, II, III
(b) I, III, IV
(c) I, II, IV
(d) none of the above statements are correct

**Answer**: **(d)** [lambda above and C (defining the "soft" vs "hard" are not the same. lamba emphasizes or de-emphasizes the regularization parameter. C is about the margin around the support vector.]

##  ET:14
Given the following paired RDDs
RDD1 = {(1, 2), (3, 4), (3, 6)}
RDD2 = {(3, 9) (3, 6)}

Using PySpark, write code to perform an inner join of these paired RDDs. What is the resulting RDD? Make your Spark available in your notebook:

A: [(3, (4, 9)), (3, (6, 9))]
B: [(3, (4, 9)), (3, (4, 6)), (3, (6, 9)), (3, (6, 6))]
C: [(3, (4, 9)), (3, (4, 6)), (3, (6, 9)), (3, (6, 9))]
D: None of the above

```python
RDD1 = sc.parallelize([(1, 2), (3, 4), (3, 6)])
RDD2 = sc.parallelize([(3, 9), (3, 6)])
print RDD1.join(RDD2).collect()
```

    [(3, (4, 9)), (3, (4, 6)), (3, (6, 9)), (3, (6, 6))]

**Answer**: **(b)** [See above]

##  ET:15
You have been tasked to build a predictive model to forecast beer sales for a chain of stores.
After doing basic exploratory analysis on the data, what is the first thing you do regarding modeling?

Select one:
(a) Construct a baseline model
(b) Determine a metric to evaluate your machine learnt models
(c) Split your data into training, validation and test subsets (or split using cross fold validatation)
(d) All of the  of the above

**Answer**: **(c) ** [Although depending on interpretation, (d) is possible as all 3 are early steps.]

##  ET:16
Use Spark and the following notebook to answer this question:

* http://nbviewer.jupyter.org/urls/dl.dropbox.com/s/6s5ph41h74bggwi/Linear-Regression-on-Beer-Data.ipynb
* https://www.dropbox.com/s/6s5ph41h74bggwi/Linear-Regression-on-Beer-Data.ipynb?dl=0

The mean absolute percentage error (MAPE), also known as mean absolute percentage deviation (MAPD), is a measure of prediction accuracy of a model for say a forecasting method in statistics,
for example in trend estimation. It usually expresses accuracy as a percentage, and is defined by the formula:

MAPE = average over all examples (100*Abs(Actual - Predicted) / Actual))

Note when Actual is zero that test row is dropped from the evaluation.

Construct a mean model for target variable `CASES18PK`. Calculate the MAPE for the mean model over the training set. Select the closest answer.

Select one:
(a) 200%
(b) 250%
(c) 20%
(d) 180%

```python
# Load data
rawSales = sc.textFile('beerSales.txt', use_unicode=False)

# Select sales18PK
sales = (rawSales
         .map(lambda x: x.split('\t'))
         .filter(lambda x: x[0] != 'Week')
         .map(lambda x: float(x[5]))
        )

sales18Pk = sales.collect()

# Mean model is just mean of cases18pk sales
meanModel = sum(sales18Pk)/len(sales18Pk)
mm = sc.broadcast(meanModel)

mape = (sales
        .map(lambda x: 100*abs(x - mm.value) / x)
       )

print mape.sum() / mape.count()
```

    200.44422333

**Answer**: **(a)** [See above]

##  ET:17
Use Spark and the following notebook to answer this question:

* http://nbviewer.jupyter.org/urls/dl.dropbox.com/s/6s5ph41h74bggwi/Linear-Regression-on-Beer-Data.ipynb
* https://www.dropbox.com/s/6s5ph41h74bggwi/Linear-Regression-on-Beer-Data.ipynb?dl=0

The target variable `CASES18P`K is skewed, so take the log of it (and make it more normally distributed) and compute the MAPE of the mean model for `CASES18PK`. Select the closest answer to your calculated MAPE.

(a) 200%
(b) 30%
(c) 20%
(d) 10%

```python
from math import log

# Convert to log
logSales = (sales
            .map(lambda x: log(x))
           )
print logSales.take(5)
```

    [6.0844994130751715, 4.584967478670572, 4.248495242049359, 3.9512437185814275, 4.1588830833596715]

```python
# Log data
logSales18Pk = logSales.collect()
meanModelLog = sum(logSales18Pk)/len(logSales18Pk)
mml = sc.broadcast(meanModelLog)

# Calculate mape
mapeLog = (logSales
           .map(lambda x: 100*abs(x - mml.value) / x)
          )

print mapeLog.sum() / mapeLog.count()
```

    19.5849342896

**Answer**: **(c)** [See above]

##  ET:18
Use Spark and the following notebook to answer this question:

* http://nbviewer.jupyter.org/urls/dl.dropbox.com/s/6s5ph41h74bggwi/Linear-Regression-on-Beer-Data.ipynb
* https://www.dropbox.com/s/6s5ph41h74bggwi/Linear-Regression-on-Beer-Data.ipynb?dl=0

Build a linear regression model using the following variables:

Log(CASES18PK)  ~  log(PRICE12PK), 	log(PRICE18PK),	log(PRICE30PK)

Please train the model on the full data and calculate  the MAPE over the same dataset and select a response from the following options that is closest this MAPE".

Select one:
(a) 4.3%
(b) 4.6%
(c) 3.5%
(d) 3.9%

```python
# Not sure why mlib regression with SGD is unstable so implemented with scikit learn (MSE is large,
# very sensitve to step size)
from pyspark.mllib.regression import LabeledPoint, LinearRegressionWithSGD, LinearRegressionModel
from math import log

data = sc.textFile('beerSales.txt')

# Load and parse the data
def parsePoint(line):
    values = [x for x in line.split('\t')]
    if values[0] != 'Week':
        return LabeledPoint(log(float(values[5])), [log(float(values[1])), log(float(values[2])), log(float(values[3]))])

parsedData = data.map(parsePoint).filter(lambda x: x != None)

# Train the model
model = LinearRegressionWithSGD.train(parsedData, intercept=True)

# Fit model
valuesAndPreds = parsedData.map(lambda p: (p.label, model.predict(p.features)))

# Evaluate model
MSE = valuesAndPreds \
    .map(lambda (v, p): (v - p)**2) \
    .reduce(lambda x, y: x + y) / valuesAndPreds.count()
print("Mean Squared Error = " + str(MSE))

# Calculate MAPE
MAPE18 = valuesAndPreds \
    .map(lambda (v,p): 100*abs(v - p)/v) \
    .reduce(lambda x, y: x + y) / valuesAndPreds.count()

print MAPE18
print model

```

    Mean Squared Error = 1.6577766584023382e+93
    8.482926161125784e+47
    (weights=[-4.863280432596992e+45,-4.639704353376874e+45,-4.401262424915796e+45], intercept=-1.6521026830999922e+45)

With scikit learn

```python
import numpy as np
import pandas as pd
from sklearn import linear_model
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

data = pd.read_table('beerSales.txt')
y = np.log(data['CASES18PK'])

columns = ['PRICE12PK', 'PRICE18PK', 'PRICE30PK']
X = pd.DataFrame(data, columns=columns)
X = np.log(X)

model = linear_model.LinearRegression(fit_intercept=True)
model.fit(X,y)
modelpred = model.predict(X)
print('Coefficients: \n', model.coef_)
# The mean squared error
print('Mean squared error: %.2f'
      % mean_squared_error(y, modelpred))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % r2_score(y, modelpred))

MAPE = sum(100*abs(y - modelpred)/y)/len(y)
# MAPE
print ('MAPE: %.2f' % MAPE )
```

    ('Coefficients: \n', array([ 2.01637714, -6.33092326,  2.47016635]))
    Mean squared error: 0.07
    Variance score: 0.93
    MAPE: 4.22

**Answer**: **(a)**

## ET:19
Recall that Spark automatically sends all variables referenced in your closures to the
worker nodes. While this is convenient, it can also be inefficient because (1) the
default task launching mechanism is optimized for small task sizes, and (2) you
might, in fact, use the same variable in multiple parallel operations, but Spark will
send it separately for each operation. As an example, say that we wanted to write a
Spark program that looks up countries by their call signs (e.g., the call sign for Ireland is EJZ) by prefix matching in an
table. In the following the "signPrefixes" variable is essentially a table with two columns "Sign" and "Country Name". The goal is
to join the following tables:

`signPrefixes` table with columns "Sign" and "Country Name"
`contactCounts` table with columns "Sign" and "count"

to yield  a new table:

`countryContactCounts` with the following columns "Country Name" and "count"

Use Spark and the following notebook to answer this question:

* http://nbviewer.jupyter.org/urls/dl.dropbox.com/s/6s5ph41h74bggwi/Linear-Regression-on-Beer-Data.ipynb
* https://www.dropbox.com/s/6s5ph41h74bggwi/Linear-Regression-on-Beer-Data.ipynb?dl=0

How can we modfify this code to make it more efficient? Choose one response only

(a) modify line 18 with `sc.broadcast(loadCallSignTable())`
(b) Use accumulators to store the counts for each country
(c) The code is already optimal
(d) none of the above

**Answer**:  **(a)**

```python

```


# Project 4, deadline  April 30

 **Molecular Dynamics project**

Date: **Spring semester 2017**

## Theoretical background and description of the physical system
In this project we will implement a model called molecular dynamics (MD). Molecular dynamics allows us to study the dynamics of atoms and explore the phase space. For those of you who are taking FYS2160 - Thermodynamics, this project will be an excellent choice. The atoms interact through a force given by the negative gradient of a potential energy function. With this force, we can integrate Newton's laws. While exploring the phase space, we will sample statistical properties of the system like energy, temperature, pressure, diffusion constant and so on.

We will also learn how to do write object oriented code. You are given
a code skeleton that contains the basic structure of an MD code, but
you will have to implement many of the functions yourself. We think
that *learn by example* is a great way to learn how to proper
object orient your code, but you can of course start from scratch if
you want. You can find the [code
here](https://github.com/andeplane/molecular-dynamics-fys3150). You
can clone or fork this repository into your own.  The class structure
which you developed for project 3 can also be used again here, as well
as the Velocity Verlet algorithm of project 3.

After you have downloaded the repository, open the project in Qt
Creator and see if it runs. The program should create 100 argon atoms
and place them uniformly inside a cubic box with sides 10
Angstroms. Each atom is given a velocity according to the
Maxwell-Boltzmann distribution so that

$$
\begin{equation}
P(v_i)d\mathrm{v}_i = \left(\frac{m}{2\pi k_B
T}\right)^{1/2} \exp\left(-\frac{m v_i^2}{2k_B T}\right)d\mathrm{v}_i,
\label{_auto1} \tag{1}
\end{equation}
$$

where $m$ is the mass of the atom, $k_B$ is
Boltzmann's constant and $T$ is the temperature. We recognize this as
a normal distribution with zero mean and standard deviation $\sigma =
\sqrt{k_B T/m}$. We recognize this also as the Boltzmann distribution $P(E) \propto \exp{(-\beta E)}$, with $E = 0.5mv^2$.
The program will evolve the system in time with no
forces so that all atoms move in a straight line. It will create a
file called *movie.xyz* containing all the timesteps ready to
visualize with for example Ovito. You can download Ovito from this [site](http://www.ovito.org/index.php/download). You should start
with the fun part - to look at the simulation in Ovito. We then
strongly recommend that, before you start, you look at the code
structure and try to understand how the different classes are
connected to each other and how a typical timestep is computed. You
can skip understanding the contents of *unitconverter.cpp* and *vec3.cpp*.

**For this project you can collaborate with fellow students and you can  hand in a common report.**
This project (together with projects 3 and 4) counts 1/3 of the final mark.

## Before starting

Before you start, we would like that you pay attention to the following two steps:
 1. First, spend some 30 minutes figuring out what molecular dynamics is and which problems it can be applied to. What areas of physics can it be used in? What about chemistry and biology? What you find here should be part of the  introductory section in your report.

 2. Then run the program, visualize the output with for example Ovito (as mentioned above). Now, spend some 30-60 minutes looking at the code and understand how the output is produced and try to understand every step in the code. The best way to do this is to start at the top of the *main()* function and follow every line and every function call, step by step. **As a part of your report, you should explain the code structure and draw a flow chart (a diagram showing how the program is executed).** This part is extremely important and will save you a lot of time. And of course, please ask us teachers if you have any questions!

### Project 5a): Getting started

When working with MD, the system size is usually limited by available computer resources. A typical large MD simulation (with a parallelized code) contains a few million atoms corresponding to a system much smaller than a cubic micron. In order to get rid of boundary effects, we usually apply periodic boundary conditions so that we simulate a system of infinite size. You need to implement the *applyPeriodicBoundaryConditions* function in the *System* class. After doing so, run the program again and notice how the atoms now are contained inside a box. Discuss the benefits of this strategy.

### Project 5b): Velocity distribution

The atoms are usually given velocities according to the Maxwell-Boltzmann distribution (you should discuss why this is the case) which will result in a nonzero net momentum in the system. Implement the *removeMomentum* function in the *System* class so that the net momentum is zero.

### Project 5c): FCC lattice

The atoms are now uniformly distributed in space. This is not very physical, so we should place the atoms in a crystal structure. When we later implement the Lennard-Jones potential, we will see that the face-centered cubic (FCC) lattice is a stable structure for the potential (it is actually the crystalline structure of argon). A lattice is built up by *unit cells* - a group of atoms - so that a larger system can be created by repeating these cells in space. An FCC lattice unit cell of size $b$ in Angstroms (this is the lattice constant) consists of four atoms with local coordinates

$$
\begin{equation}
	\mathbf{r}_1 = 0 \hat{\mathbf{i}} + 0 \hat{\mathbf{j}} + 0 \hat{\mathbf{k}},
\label{_auto2} \tag{2}
\end{equation}
$$

$$
\begin{equation}
	\mathbf{r}_2 = \frac{b}{2} \hat{\mathbf{i}} + \frac{b}{2} \hat{\mathbf{j}} + 0 \hat{\mathbf{k}},
\label{_auto3} \tag{3}
\end{equation}
$$

$$
\begin{equation}
	\mathbf{r}_3 = 0 \hat{\mathbf{i}} + \frac{b}{2} \hat{\mathbf{j}} + \frac{b}{2} \hat{\mathbf{k}},
\label{_auto4} \tag{4}
\end{equation}
$$

$$
\begin{equation}
	\mathbf{r}_4 = \frac{b}{2} \hat{\mathbf{i}} + 0 \hat{\mathbf{j}} + \frac{b}{2} \hat{\mathbf{k}}.
\label{_auto5} \tag{5}
\end{equation}
$$

You can now create $N_x \times N_y \times N_z$ such unit cells next to each other to form a larger system so that the origin of unit cell $(i,j,k)$ is

$$
\begin{equation}
	\mathbf{R}_{i,j,k} = i \hat{\mathbf{u}}_1 + j \hat{\mathbf{u}}_2 + k \hat{\mathbf{u}}_3,
\label{_auto6} \tag{6}
\end{equation}
$$

where $i=0,1,..., N_x-1, j=0,1,..., N_y-1, k=0,1,..., N_z-1$. The unit vectors of the unit cells are scaled with the lattice constant $b$ so that

$$
\begin{equation}
	\hat{\mathbf{u}}_1 = b\hat{\mathbf{i}}, \quad \hat{\mathbf{u}}_2 = b\hat{\mathbf{j}}, \quad \hat{\mathbf{u}}_3 = b\hat{\mathbf{k}}.
\label{_auto7} \tag{7}
\end{equation}
$$

Implement the function *createFCCLattice(int numberOfUnitCellsEachDimension, double latticeConstant)* in the *System* class. Remember to remove the 100 atoms that are created as the code is right now. Use lattice constant $b=5.26$ $\mathrm{Angstrom}$. Remember to update the system size so the *applyPeriodicBoundaryConditions* function works properly. If you now visualize the result of the program, you should see the nice crystalline structure in the beginning of the simulation. What is the density $\rho$ in your system?

### Project 5d): Calculation of forces

While this looks nice, we need to implement the computation of forces in order to see interesting physics. In this project, we will use the [Lennard-Jones potential](http://en.wikipedia.org/wiki/Lennard-Jones_potential)  which calculates the energy between two atoms $i$ and $j$ as

$$
\begin{equation}
	U(r_{ij}) = 4\epsilon\left[\left(\frac{\sigma}{r_{ij}}\right)^{12} - \left(\frac{\sigma}{r_{ij}}\right)^6\right],
\label{_auto8} \tag{8}
\end{equation}
$$

where $r_{ij} = \vert\mathbf{r}_i - \mathbf{r}_j\vert$ is the distance from atom $i$ to atom $j$, $\epsilon$ is the depth of the potential well (with dimension energy) and $\sigma$ is the distance at which the potential is zero. For argon, optimal values of the parameters are:

$$
\begin{equation}
	\frac{\epsilon}{k_B} = 119.8\mathrm{K}, \sigma=3.405 \mathrm{Angstrom}.
\label{_auto9} \tag{9}
\end{equation}
$$

In our code, we use the so called MD units (see the units definition at the end). This potential reproduces thermodynamic equilibrium properties that are in good agreement with experimental values for argon. The equation of state for this system is the van der Waals equation of state. Phases such as solid, liquid and gas are reproduced from this simple model with phase transitions and other properties as well with a  remarkable agreement with data.

The total potential energy $V$ in the system is computed by summing over all pairs of atoms (counting each pair only once)

$$
\begin{equation}
	V = \sum_{i>j} U(r_{ij}).
\label{_auto10} \tag{10}
\end{equation}
$$

The force is calculated by taking the negative gradient of the potential

$$
\begin{equation}
	\mathbf{F}(r_{ij}) = -\nabla U(r_{ij}),
\label{_auto11} \tag{11}
\end{equation}
$$

giving the $x$-component (the other components are calculated the same way)

$$
\begin{equation}
	F_x(r_{ij}) = -\frac{\partial U}{\partial r_{ij}}\frac{\partial r_{ij}}{\partial x_{ij}},
\label{_auto12} \tag{12}
\end{equation}
$$

where $x_{ij}$ is the $x$-component of $\mathbf{r}_{ij}$.

Calculate (analytically) the force and implement the *calculateForces* function in the LennardJones class, using the Velocity Verlet algorithm that you developed for project 3. You will have to take care of the periodic boundary conditions. You could use the minimum image convention as described [here](http://en.wikipedia.org/wiki/Periodic_boundary_conditions). Now run the simulation with 5 unit cells in each dimension (500 atoms) for different initial temperatures. Can you tell if the system is in a solid state just by looking at the system in Ovito? At what initial temperature is the crystal melting?

### Project 5e): Kinetic and potential energies

We now have a working Molecular dynamics code! The next step is to calculate some physical properties of the system. The easiest properties to measure are the kinetic and potential energy (calculated in the LennardJones class). The kinetic energy is defined as

$$
\begin{equation}
	E_k = \sum_{i=1}^{N_\text{atoms}} \frac{1}{2} m_i v_i^2,
\label{_auto13} \tag{13}
\end{equation}
$$

where $m_i$ and $v_i$ is the mass and the speed of atom $i$. You can use the function *atom->velocity.lengthSquared()* to calculate the dot product of the velocities.

You can also calculate an estimate of the temperature through the equipartition theorem, see for example D. Schroeder's *An Introduction to Thermal Physics* for details,

$$
\begin{equation}
	\langle E_k \rangle = \frac{3}{2}N_\text{atoms} k_B T.
\label{_auto14} \tag{14}
\end{equation}
$$

We can use this to define an *instantaneous* temperature

$$
\begin{equation}
	T = \frac{2}{3}\frac{E_k}{N_\text{atoms} k_B}.
\label{_auto15} \tag{15}
\end{equation}
$$

All of these quantities should be calculated in the *StatisticsSampler* class. All the statistical quantities should be saved to a file which needs to be implemented in the *saveToFile* function in the same class.

As you will see, since the temperature is proportional to the kinetic energy, its initial value will drop in the beginning when the system is initialized in the crystal structure. Why does this happen? What happens to the total energy? In order to have a system with final temperature $T$, what initial temperature $T_i$ is needed? What is the ratio $T/T_i$? Do multiple simulations with different random seeds and plot this ratio as a function of time. Again visualize with for example *Ovito* and try to determine at what temperature the system actually melts at. After the temperature drops, it will reach a more or less stable value. The system is then said to be in an equilibrium state.

### Project 5f): Diffusion constant

The last exercise is to compute the diffusion constant and use this to measure the melting temperature. We use the Einstein relation that relates the so called mean square displacement (MSD) to the diffusion constant

$$
\begin{equation}
	\langle r^2(t) \rangle = 6Dt,
\label{_auto16} \tag{16}
\end{equation}
$$

where $D$ is the diffusion constant, $t$ is time. The mean square displacement for atom $i$ is calculated as

$$
\begin{equation}
	r_i^2(t) = |\mathbf{r}_i(t) - \mathbf{r}_i(0)|^2,
\label{_auto17} \tag{17}
\end{equation}
$$

where $\mathbf{r}_i(t)$ is the position of atom $i$ at time $t$. When you implement this you need to add the initial position as a new property in the *Atom* class. You also need to update the *applyPeriodicBoundaryConditions* so that $r_i^2(t)$ is the actual displacement as if no periodic boundary condition has occured. Why is this important?

We can use the diffusion constant to find the melting temperature. Atoms in a solid will not diffuse much, meaning that a solid will have a diffusion constant close to zero. Now solve the Einstein relation for the diffusion constant $D$ and add a function in *StatisticsSampler* thats measures this quantity. Plot the diffusion constant as a function of temperature $T$ and use this to find an estimate for the melting temperature. Use what you found earlier  to make sure you simulate temperatures both above and below the melting temperature. Also remember not to use the initial temperature, but the measured temperature after the system has reached equilibrium. You can use the builtin *UnitConverter* to convert the temperature to SI units as illustrated in the top of the *main* function.

# What is really going on here?
The idea of an MD simulation is to sample microstates from a statistical ensemble. In our simulation, we have a constant number of atoms $N$, a constant volume $V$ and a (more or less, depending on our integrator) constant energy $E$. This corresponds to the microcanonical ensemble (NVE).

Although we produce the dynamics of atoms, we should not see MD as a model that will give the true trajectories of the atoms, but rather a model that allows us to explore the phase space according to the probabilities we can calculate with statistical mechanics. A fundamental assumption in any MD simulation is the ergodic hypothesis. It states that *over long periods of time, the time spent by a system in some region of the phase space of microstates with the same energy is proportional to the volume of this region, i.e., that all accessible microstates are equiprobable over a long period of time*. This means that by evolving the atoms through time will visit microstates of the ensemble according to the true probabilities given by the ensemble.

# Units

The program uses by default a set of units. We have chosen the following four units

$$
\begin{equation}
	\text{1 unit of mass } = 1 \text{ a.m.u} = 1.661\times 10^{-27}\mathrm{kg},
\label{_auto18} \tag{18}
\end{equation}
$$

$$
\begin{equation}
	\text{1 unit of length } = 1.0 \mathrm{angstrom} = 1.0\times 10^{-10}\mathrm{m},
\label{_auto19} \tag{19}
\end{equation}
$$

$$
\begin{equation}
	\text{1 unit of energy } = 1.651\times 10^{-21}\mathrm{J},
\label{_auto20} \tag{20}
\end{equation}
$$

$$
\begin{equation}
	\text{1 unit of temperature} = 119.735\mathrm{K}.
\label{_auto21} \tag{21}
\end{equation}
$$

Boltzmann's constant is then equal to one (convince yourself about this), and other units can be derived by using known relations. We can for example find the unit of time by using $E=mc^2$. We have that

$$
\begin{equation}
	\text{Energy} = \text{Mass} \times \frac{\text{Length}^2}{\text{Time}^2},
\label{_auto22} \tag{22}
\end{equation}
$$

which can be solved for time

$$
\begin{equation}
	\text{Time} = \text{Length} \times \sqrt{\frac{\text{Mass}}{\text{Energy}}}.
\label{_auto23} \tag{23}
\end{equation}
$$

By inserting the values above, we get

$$
\begin{equation}
	\text{1 unit of time } = 1.0 \times 10^{-10}\sqrt{\frac{1.661\times 10^{-27}}{1.651\times 10^{-21}}} \text{ s} = 1.00224\times 10^{-13}\mathrm{s}.
\label{_auto24} \tag{24}
\end{equation}
$$

This way, we can continue working out the values of the other units. What are the units of force? And pressure? In the project, a class that calculates these values is given.

## Introduction to numerical projects

Here follows a brief recipe and recommendation on how to write a report for each
project.

  * Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  * Include the source code of your program. Comment your program properly.

  * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  * Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  * Try to give an interpretation of you results in your answers to  the problems.

  * Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  * Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.

## Format for electronic delivery of report and programs

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

  * Use your github repository to upload your report. Indicate where the report is by creating for example a **Report** folder. Please send us as soon as possible your github username.

  * Place your programs in a folder called for example **Programs** or **src**, in order to indicate where your programs are. You can use a README file to tell us how your github folders are organized.

  * In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.

  * In this and all later projects, you should include tests (for example unit tests) of your code(s).

  * Comments  from us on your projects, with score and detailed feedback will be emailed to you.

Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. You can then hand in a common report.


```python
%load_ext autoreload
%autoreload 2
```

```python
import math
import numpy as np
import pandas as pd
import pymc3 as pm
import seaborn as sns
from tqdm.notebook import tqdm

import scipy.stats as scs
from scipy.special import comb
from statsmodels.stats.proportion import proportions_ztest

import how_to_ab_test.sample_data as sample_data
import how_to_ab_test.validation as validation
```

# How to AB Test

This lab explores how one goes about AB testing.

# Tests

### Fisher Exact Test

The Fisher exact test looks at the probability of this or more extreme a skew of success rates in a randomly drawn 2x2 contingency table conditioned on having the same marginals.

Let

$N$    be the total number of draws;

$n$    the number of draws in Group 0;

$K$    the total number of successes;

Then, conditioning on the marginals leaves a single degree of freedom. This can, WLOG, be taken to be

$k$    the number of successes in Group 0;

Then, more extreme cases are $k > k_0$. (With the corresponding other $k < K-k_0$ for the two-sided test.)

For the probability of this table being drawn ($k = k$) given the marginals, consider the analogy to drawing without replacement $k$ balls of type 1 and $n-k$ of type 2 from an urn with $K$ of type 1 and $N-K$ of type 2.

To calculate this, we want the number of ways to draw the first row with the given split ($k$ out of $K$ from type 1 and $n-k$ from $N-K$) out of the possible ways to draw the first row ($n$ from a total $N$). (Note - having picked the first row, that fixes the whole table.)

So we get the hypergeometric distribution:
$$
P(t_{0,0} = k) = \frac{{K \choose k}{N-K \choose n-k}}{{N \choose n}}
$$

For the p-value of the Fisher exact test, we then just sum over values of $k$ that are as or more extreme than the observed one.

Implemented in `scipy.stats.fisher_exact`:

```python
def fisher_exact(table):
    N = table.sum().sum()
    n = table.sum(axis=1)[0]
    K = table.sum(axis=0)[0]
    k = table.iloc[0, 0]

    num = 0
    for ki in range(k, K+1):
        num += comb(K, ki, exact=True) * comb(N-K, n-ki, exact=True)

    return num / comb(N, n, exact=True)

for table in sample_data.random_contingency_tables_2x2(10, is_equal_p=True):
    odds_ratio, pvalue = scs.fisher_exact(table, alternative='greater')
    pvalue_ = fisher_exact(table)
    # print((pvalue, pvalue_))
    assert abs(pvalue-pvalue_)<1e-8
```

Follow Up: It looks very Bayes-like. Is there any interpretation along those lines?

### Large Sample Z Test

The z test is a large-sample approximation that the proportions in two groups are equal.

It uses the fact that, by the CLT (and in practice converging very quickly - see below), the natural estimator for the group proportion is roughly normal:

$$
\hat{p} := \frac{X}{n} \sim \mathcal{N}\Big(p, \frac{p(1-p)}{n}\Big)
$$

where $ X \sim Binom(p, n)$.

Note - It is a normal distribution, not a t. I haven't found a reference for this. Beyond that we are talking about large-sample (and so the t-test should be approximately normal anyways), I suspect that we have a normal distribution because the binomial is a single-parameter distribution so the extra degree of freedom for the SD doesn't exist in this case.

To test the null hypothesis that the two groups come from the same distribution, we consider:

$$
z := \hat{p_1} - \hat{p_2} \sim \mathcal{N}\big(0, \hat{s}^2 \big)
$$

where

$$
\begin{align}
\hat{s}^2 &= Var(\hat{p}) \\
    &= Var(\hat{p_1}) + Var(\hat{p_2}) \\
    &= \frac{\hat{p_1}(1-\hat{p_1})}{n_1} + \frac{\hat{p_2}(1-\hat{p_2})}{n_2}
\end{align}
$$

called the Satterthwaite approximation or we can pool, under the null hypothesis, the groups when estimating the SD, giving the pooled approximation

$$
\begin{align}
\hat{s}^2 &= Var(\hat{p}) \\
    &= Var(\hat{p_1}) + Var(\hat{p_2}) \\
    &= \frac{\hat{p}(1-\hat{p})}{n_1} + \frac{\hat{p}(1-\hat{p})}{n_2} \\
    &= \hat{p}(1-\hat{p}) \Big( \frac{1}{n_1} + \frac{1}{n_2} \Big)
\end{align}
$$

where

$$
\hat{p} = \frac{n_1 p_1 + n_2 p_2}{n_1 + n_2} = \frac{x_1 + x_2}{n_1 + n_2}
$$

The usual one-sided or two-sided tests are then as usual.

Implemented in `statsmodels.stats.proportion.proportions_ztest`:

```python
def proportions_ztest_(table, *, pooled_approx=False):
    assert table.shape == (2, 2)

    s1, s2 = table.iloc[:, 0]
    n1, n2 = table.sum(axis=1)

    p1, p2 = s1 / n1, s2 / n2

    if pooled_approx:
        p = (s1 + s2) / (n1 + n2)
        v = p * (1 - p) * (1 / n1 + 1 / n2)
    else:
        v = p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2

    if not v > 0:
        return np.nan, np.nan

    z_stat = (p1 - p2) / np.sqrt(v)

    return z_stat, scs.norm.sf(np.abs(z_stat)) * 2

for table in sample_data.random_contingency_tables_2x2(10, is_equal_p=True):
    z_stat, pvalue = proportions_ztest(
        table.Success,
        table.Success+table.Fail,
        alternative='two-sided',
        prop_var=False
    )
    z_stat_, pvalue_ = proportions_ztest_(table, pooled_approx=True)
    # print((pvalue, pvalue_))
    assert (np.isnan(pvalue) and np.isnan(pvalue_)) or abs(pvalue-pvalue_)<1e-8
```

### Chi-Squared Test

This is a large-sample approximation that two multinominial distribution match. Note that, unlike the Fisher and Z tests, it is limited to 2-sided tests but applies for any number of categories and groups.

Given a multinomial sample

$$
\begin{align}
    X &\sim Multinomial_k(n, \pi) \\
    P(X=x) &= \frac{n!}{x_1!...x_k!} \pi_1^{x_1} ... \pi_k^{x_k}
\end{align}
$$

it is shown that under $ H_0: \pi = \pi_0 $ we have

$$
X^2 = \sum_{i=1}^k \frac{(X_i - E_i)^2}{E_i} \sim \chi^2(k-1)
$$

where $ E_i = X_i * \pi_i $.

Extending to $g$ groups, we have

$$
X^2 = \sum_{cells} \frac{(X_i - E_i)^2}{E_i} = \sum_{i=1}^{k} \sum_{j=1}^g \frac{(X_i^{(g)}-E_i)^2}{E_i} \sim \chi^2\big((k-1)(g-1)\big)
$$

where $E_i$ is estimated by the category totals $ E_i = \sum_{j=1}^g X_i^{(g)}$

Because the test is applying a discrete observation against a continuous approximation, Yates proposed a continuity correction. This subtracts 0.5 from each absolute error before squaring. Subsequent authors have proposed other corrections.

Note: Cressie & Read extended this by remarking that this and a few other tests were related as special cases of a continuous family of the "power divergence" statistics:

$$
2n I^\lambda = \frac{2}{\lambda (\lambda + 1)} \sum_{i=1}^k X_i\Bigg( \Big(\frac{X_i}{E_i}\Big)^\lambda - 1 \Bigg)
$$

Note: This is a square of the normal approximation. So it (the uncorrected version) is exactly equivalent to the two-sided Z-test.

Implemented in `scipy.stats.chi2_contingency`:

```python
def chi2_contingency(table):
    by_category = table.values.sum(axis=0, keepdims=True)
    pi = by_category/by_category.sum()
    by_group = table.values.sum(axis=1, keepdims=True)

    E = by_group*pi

    X2 = (np.square(table.values-E)/E).sum()

    dof = (table.shape[0]-1)*(table.shape[1]-1)

    return X2, 1.0 - scs.chi2.cdf(X2, dof), dof, E

n = 10
n_valid = 0
for table in sample_data.random_contingency_tables_2x2(n, is_equal_p=True):
    if not (table.sum(axis=0) == 0).any():
        # chi2_contingency invalid if any category has 0 observed
        n_valid += 1
        chi2, pvalue, dof, expected = scs.chi2_contingency(table.values, correction=False)
        chi2_, pvalue_, dof_, expected_ = chi2_contingency(table)
        # print((pvalue, pvalue_))
        assert abs(pvalue-pvalue_)<1e-8
# print((n_valid, n))
assert n_valid > n/2
```

### Bayes

The Bayes approach, naturally, is rather different.

We get a posterior distribution for the probabilities (say $p_A$ and $p_B$). From this, we have a few options:

1. Look at probabilities (eg $P(p_A > p_B)$)
2. Look at credible intervals (eg Highest Posterior Density region (HPD): ie the $100*(1-\alpha)$% HPD is the $R(f_\alpha) := \{x: f(x) > f_\alpha\}$ where $f_\alpha$ is the maximal such that $ P(R(f_\alpha)) >= \alpha$)
3. Look at an expected expected loss and apply a decision rule.

Indeed (up to chosing the prior) Bayes formula gives the full answer and it's up to the user to choose a decision rule. In this sense, this is the true answer. The only real downside is the computational difficulty. The real question is how the frequentist approaches compare.

```python
def bayes_2x2_contingency(table):
    n1, n2 = table.values.sum(axis=1)
    s1, s2 = table.iloc[:, 0]

    with pm.Model() as model:
        p1 = pm.Beta('p1', alpha=1, beta=1)
        p2 = pm.Beta('p2', alpha=1, beta=1)
        pm.Binomial('s1', n=n1, p=p1, observed=s1)
        pm.Binomial('s2', n=n2, p=p2, observed=s2)

        p_diff = pm.Deterministic('p_diff', p2-p1)

        trace=pm.sample(2000, tune=2000)

    return trace

table = sample_data.random_contingency_table_2x2()
print(table)

trace = bayes_2x2_contingency(table)
pm.traceplot(trace, var_names=['p_diff'])
pm.plot_posterior(trace, var_names=['p_diff'], ref_val=0.0)
pm.summary(trace)
```

# Validation

The key question is how the frequentist approximations hold up. In particular,

1. How close do the approximations match their intended significance levels?
2. How do their powers compare?

This will all be done in the context of 2-group / 2-outcome A/B tests, as this is common case across all the investigated tests.

To answer the first question, take $p_1 = p_2$ and consider the number of rejections vs $\alpha$ across a range of parameters:

```python
rejections_type1 = pd.concat([
    validation.rejections(n=n, p1=p, p2=p, n_samples=10000, alphas=[0.5, 0.1, 0.05, 0.025, 0.01], exclude=[]).assign(n=n, p=p)
    for n in tqdm([10, 100, 1000], leave=False)
    for p in tqdm([0.01, 0.1, 0.25, 0.5, 0.9], leave=False)
])
```

    HBox(children=(FloatProgress(value=0.0, max=3.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))

```python
g = sns.FacetGrid(data=rejections_type1, row='n', col='test')
g.map_dataframe(validation.rejection_heatmap_by(row='p', col='alpha', diff_to_alpha=True))
```

From which the broad expected trends are observed:

* The accuracy improves as $n$ increases
* The accuracy improves as the confidence interval shrinks
* By $n = 1000$, all methods show high accuracy
* The uncorrected $\chi^2$ test is exactly the same as the z-test so omitted.

Some unexpected observations:

* The corrected $\chi^2$ test shows Type I error profile very similar to that of the Fisher test
* The Fisher (and corrected $\chi^2$ tests) seem less accurate for lower $n$ and the error is uniformly on the low side

These are addressed in https://www.jstor.org/stable/1164680 and https://www.jstor.org/stable/2681839. Confirming that indeed the corrected chi^2 for a 2x2 table with equal numbers in both groups approximates the Fisher exact tests.

To get a feel for the second question, consider the small-data case $n = 10$ across different values of the underlying probability and different factors by which the second group probability exceeds the first:

```python
rejections_type2 = pd.concat([
    validation.rejections(n=10, p1=p, p2=p*factor, n_samples=10000, alphas=[0.5, 0.1, 0.05, 0.025, 0.01], exclude=[]).assign(p=p, factor=factor)
    for factor in tqdm([1.05, 1.1, 1.5], leave=False)
    for p in tqdm([0.01, 0.1, 0.25, 0.5], leave=False)
])
```

    HBox(children=(FloatProgress(value=0.0, max=3.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=4.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=4.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=4.0), HTML(value='')))

```python
g = sns.FacetGrid(data=rejections_type2, row='factor', col='test')
g.map_dataframe(validation.rejection_heatmap_by(row='p', col='alpha'))
```

Again, the Fisher test compares very similarly to the corrected $\chi^2$ test. And again the z-test performs better on this metric. Though the results aren't exactly encouraging in any case - even where the second group probability is 50% greater than that of the first group, $H_0$ can only be rejected in about 20% of the cases.

# Conclusions

Generally, the results support that the z-test (similarly uncorrected $\chi^2$ if doing a 2-sided test) performs closer it's stated accuracy and gives higher power.

But ... See https://www.jstor.org/stable/1164680 and https://www.jstor.org/stable/2681839 for a discussion around the validity of the validation procedure. Particularly, these results depend very much on the underlying generation process assumed. Even in this frequentist framework, it's impossible to escape this model selection issue.

# References

### Tests

https://www.itl.nist.gov/div898/handbook/prc/section3/prc33.htm

https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Tests_for_Two_Proportions.pdf

https://www.jstor.org/stable/2983604

https://towardsdatascience.com/the-math-behind-a-b-testing-with-example-code-part-1-of-2-7be752e1d06f

https://www.jstor.org/stable/2345686

https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf

https://www.jstor.org/stable/1164680

https://www.jstor.org/stable/2681839

### Methodology

http://blog.analytics-toolkit.com/2017/statistical-significance-ab-testing-complete-guide/

https://www.analytics-toolkit.com/pdf/Efficient_AB_Testing_in_Conversion_Rate_Optimization_-_The_AGILE_Statistical_Method_2017.pdf

https://www.evanmiller.org/how-not-to-run-an-ab-test.html

https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html


# Exact Diagonalization of a 1D Spin Chain Part 2
<b> Christina Lee

Category: Grad

Prerequisites: Many-Body Quantum Mechanics, Part 1 </b>

Check out part 1 for the theoretical background.  Today is all programming.

Remember to keep a reasonable number of spins, $n$.  The number of states goes as $2^n$, and the size of the Hamiltonian will go as $2^n \times 2^n$.  A 10-spin chain will have over a million entries, before taking into account any memory reducing tricks.

Here's the details of the largest spin-chains that fit on my machine which has 16GB of RAM and a 3.2 GHz Intel i5 processor:

| n | m_z | t (min) | Memory (Gb) |
|---|-----|---------|-------------|
|16 | 8   | 1.5     | 2           |
|18 | 9   | ?       |  >16        |
|18 | 8   | 32.1    |   15.5      |

I have included a file in this directory, ED.jl, that is just the necessary executable parts of this Jupyter notebook.  For large $n$, I recommend running ED.jl.

```julia
using LinearAlgebra
```

Here, we input one parameter `n`, the number of spins in our chain.

The program automatically calculates the parameter `nstates`.

```julia
n=4
nstates=2^n
```

    16

Now, let's write out all of our possible states in the $S^z$ basis.

```julia
psi=convert.(Int8, collect(0:(nstates-1)) )
for p in psi
    println(bitstring(p)[end-n:end],' ',p)
end
```

    00000 0
    00001 1
    00010 2
    00011 3
    00100 4
    00101 5
    00110 6
    00111 7
    01000 8
    01001 9
    01010 10
    01011 11
    01100 12
    01101 13
    01110 14
    01111 15

As in Part 1, we will be using the powers of 2 to compute magnetization, and masks to flip spins.  To not have to calculate them each time, we just store them in memory.

```julia
powers2=collect(0:(n-1));
powers2=convert.(Int8, 2.0 .^powers2 );

mask=[0;powers2]+[powers2;0];
mask=convert.(Int8, [mask[2:end-1];[1+2^(n-1)]])

for m in mask
    println(bitstring(m)[end-n:end])
end
```

    00011
    00110
    01100
    01001

In Part 1, I used the number of up-spins as a proxy for magnetization.  Here, we need the <i>actual</i> magnetization, not a proxy.  An up-spin is $+1/2$ and a down-spin is $-1/2$.  We modify our magnetization by

\begin{equation}
m=\frac{1}{2}\left(n_{\uparrow}-n_{\downarrow} \right)
= \frac{1}{2} \left( n_{\uparrow}- \left( n - n_{\uparrow} \right) \right),
\end{equation}

\begin{equation}
m = n_{\uparrow} - \frac{n}{2}.
\end{equation}

```julia
m=zeros(Int8,length(psi))
for i in 1:nstates
    m[i]=sum((psi[i].&powers2)./(powers2))
end
m=m.-n/2
```

    16-element Array{Float64,1}:
     -2.0
     -1.0
     -1.0
      0.0
     -1.0
      0.0
      0.0
      1.0
     -1.0
      0.0
      0.0
      1.0
      0.0
      1.0
      1.0
      2.0

## Grouped by Magnetization

Now that we have the magnetizations corresponding to each state, we perform some trickery that allows us to reduce the difficulty of our problem dramatically.

Magnetization is a conserved quantity. By Noether's theorem, we know that the Hamiltonian is not going to mix states of different magnetizations. We only deal with one magnetization at a time, which is a much smaller problem.

```julia
# The possible values for magnetization
ma=collect(0:1:n).-n/2
```

    5-element Array{Float64,1}:
     -2.0
     -1.0
      0.0
      1.0
      2.0

Now let's just pick out a single magnetization quantum number $mz$ and only work with that matrix for the rest of the post.

```julia
# The magnetic quantum number
mz=3

# An array of states with the correct magnetization
psi_mz=psi[m.==ma[mz]]

[psi_mz bitstring.(psi_mz) m[psi_mz.+1]]
```

    6×3 Array{Any,2}:
      3  "00000011"  0.0
      5  "00000101"  0.0
      6  "00000110"  0.0
      9  "00001001"  0.0
     10  "00001010"  0.0
     12  "00001100"  0.0

And now creating the matrix.

```julia
dim=length(psi_mz)

M=ma[mz]*(ma[mz]+1)*Matrix{Float64}(I,dim,dim)
#M=zeros(Float64,dim,dim); use this for XY model
```

    6×6 Array{Float64,2}:
     0.0  0.0  0.0  0.0  0.0  0.0
     0.0  0.0  0.0  0.0  0.0  0.0
     0.0  0.0  0.0  0.0  0.0  0.0
     0.0  0.0  0.0  0.0  0.0  0.0
     0.0  0.0  0.0  0.0  0.0  0.0
     0.0  0.0  0.0  0.0  0.0  0.0

Though we have significantly reduced the size of the matrix by restricting to one magnetization, we no longer have our states in `1,2,3,4...` order.  Their position in an array no longer corresponds to their value.  Therefore, we need a function to determine their index once we know their value.

We can find the index of the flipped state multiple different ways, but the simplest is by the Midpoint method.  We split the interval in half, and see if the value we are looking for is higher or lower than the middle point.  Then we get a new interval.

```julia
function findstate(state,set)

    #Lower bound of interval
    imin=1
    #Upper bound of interval
    imax=length(set)

    # checking if the lower bound is what we are looking for
    if set[imin] == state
        return imin
    end
    # checking if the upper bound is what we are looking for
    if set[imax] == state
        return imax
    end

    # Initializing variables
    # looking to see if we've found it yet
    found=false
    # how many times we've gone around the while loop
    count=0

    while found==false && count < length(set)

        count+=1
        tester=floor(Int,imin+(imax-imin)/2)

        if state < set[tester]
            imax=tester-1
        elseif state > set[tester]
            imin=tester+1
        else
            found=true
            return tester
        end
    end

    if found == false
        println("findstate never found a match")
        println("Are you sure the state is in that Array?")
    end

    return 0
end
```

    findstate (generic function with 1 method)

Now time to generate the matrix.

For each state and for each pair of adjacent spins within that state, we apply the operator that flips adjacent spins, `$mask`.  Sometimes the adjacent spins will take on the same value, `00` or `11`.  In this circumstance, the off-diagonal part of the Hamiltonian would not act on those spins.  The state generated by the operator would have a different magnetization, and we can neglect that pair.

 If the new state produced by this process has the same magnetization, we know that the flip exists in the Hamiltonian, and add the entry to the matrix.

In this algorithm, we do end up going over each pair twice, but I have yet to figure out how to take advantage of the degeneracy to cut the calculation in half.  Let me know if you have a better way to write this.

```julia
mp=sum(psi_mz[1].&powers2./powers2)

for ii in 1:length(psi_mz)
    p=psi_mz[ii]
    for jj in 1:n
        flipped=p⊻mask[jj]
        if sum((flipped.&powers2)./powers2) == mp
            tester=findstate(flipped,psi_mz)
            M[ii,tester]=.5
            M[tester,ii]=.5
            println(bitstring(p)[end-n:end],'\t',bitstring(flipped)[end-n:end])
        end

    end
end
```

    00011	00101
    00011	01010
    00101	00110
    00101	00011
    00101	01001
    00101	01100
    00110	00101
    00110	01010
    01001	01010
    01001	00101
    01010	01001
    01010	01100
    01010	00110
    01010	00011
    01100	01010
    01100	00101

```julia
M
```

    6×6 Array{Float64,2}:
     0.0  0.5  0.0  0.0  0.5  0.0
     0.5  0.0  0.5  0.5  0.0  0.5
     0.0  0.5  0.0  0.0  0.5  0.0
     0.0  0.5  0.0  0.0  0.5  0.0
     0.5  0.0  0.5  0.5  0.0  0.5
     0.0  0.5  0.0  0.0  0.5  0.0

```julia
F=eigen(M)
display(F.values)
display(F.vectors)
```

    6-element Array{Float64,1}:
     -1.4142135623730943
      0.0
      0.0
      0.0
      1.9984014443252818e-15
      1.414213562373095

    6×6 Array{Float64,2}:
      0.353553   0.0       -0.211325   0.788675  -0.288675    -0.353553
     -0.5        0.707107   0.0        0.0       -5.1279e-16  -0.5
      0.353553   0.0        0.788675  -0.211325  -0.288675    -0.353553
      0.353553   0.0       -0.57735   -0.57735   -0.288675    -0.353553
     -0.5       -0.707107   0.0        0.0       -5.1279e-16  -0.5
      0.353553   0.0        0.0        0.0        0.866025    -0.353553

Now we have eigenvalues and eigenvectors! You just solved the Heisenburg Spin Chain!

In my next post, I will analyze what this tells us about the system and what we can do with the information.

@inproceedings{sandvik2010computational,
  title={Computational studies of quantum spin systems},
  author={Sandvik, Anders W},
  booktitle={AIP Conference Proceedings},
  volume={1297},
  number={1},
  pages={135--338},
  year={2010},
  organization={AIP}
}

```julia

```


## Linear Regression in Python for Engineers, Data Scientists and Geoscientists
### Michael Pyrcz, Associate Professor, University of Texas at Austin

#### Contacts: [Twitter/@GeostatsGuy](https://twitter.com/geostatsguy) | [GitHub/GeostatsGuy](https://github.com/GeostatsGuy) | [www.michaelpyrcz.com](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446)

This is a tutorial / demonstration of **Linear Regression**.  In $Python$, the $SciPy$ package, specifically the $Stats$ functions (https://docs.scipy.org/doc/scipy/reference/stats.html) provide excellent tools for efficient use of statistics.
I have previously provided this example in R and posted it on GitHub:

1. R https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.R
2. Rmd with docs https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.Rmd
3. knit as an HTML document(https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.html)

In all cases, I use the same dataset available as a comma delimited file (https://git.io/fxMql).

This tutorial includes basic, calculation of a linear regression model (only 1 predictor and 1 response), testing the significance of the parameters, calculation the parameter confidence intervals and the conditional prediction interval.

##### Caveats

I have not included all the details, specifically the test assumptions in this document.  These are included in the accompanying course notes, Lec09_Bivariate_QQ_Regres.pdf.

#### Project Goal

0. Introduction to Python in Jupyter including setting a working directory, loading data into a Pandas DataFrame.
1. Learn the basics for working with linear regresion in Python.
2. Demonstrate the efficiency of using Python and SciPy package for statistical analysis.

#### Load the required libraries

The following code loads the required libraries.

```python
import os                                                   # to set current working directory
import numpy as np                                          # arrays and matrix math
import scipy.stats as st                                    # statistical methods
import pandas as pd                                         # DataFrames
import matplotlib.pyplot as plt                             # for plotting
import math                                                 # for square root
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see below) data file in this working directory.

```python
#os.chdir("C:\PGE337")                                  # set the working directory
```

#### Loading Data

Let's load the provided dataset. 'Density_Por_data.csv' is available at https://github.com/GeostatsGuy/GeoDataSets. It is a comma delimited file with 20 porosity measures from 2 rock units from the subsurface, porosity (as a fraction). We load it with the pandas 'read_csv' function into a data frame we called 'df' and then preview it by printing a slice and by utilizing the 'head' DataFrame member function (with a nice and clean format, see below).

```python
#df = pd.read_csv("Density_Por_data.csv")                    # read a .csv file in as a DataFrame
df = pd.read_csv(r"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/Density_Por_data.csv") # load data from Dr. Pyrcz's GitHub repository
df.head()                                                   # we could also use this command for a table preview
```

      Density
      Porosity

      0
      1.281391
      16.610982

      1
      1.404932
      13.668073

      2
      2.346926
      9.590092

      3
      1.348847
      15.877907

      4
      2.331653
      4.968240

It is useful to review the summary statistics of our loaded DataFrame.  That can be accomplished with the 'describe' DataFrame member function.  We transpose to switch the axes for ease of visualization.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Density
      105.0
      1.737917
      0.288278
      0.996736
      1.552713
      1.748788
      1.906634
      2.410560

      Porosity
      105.0
      12.531279
      3.132269
      4.966421
      10.546483
      12.411608
      14.230930
      20.964941

Here we extract the X1 and X2 unit porosity samples from the DataFrame into separate arrays called 'X1' and 'X2' for convenience.

```python
por = df['Porosity']
den = df['Density']
denv = np.linspace(1.0,2.4,100)
```

#### Linear Regression Model

Let's first calculate the linear regression model

```python
linear = st.linregress(den,por)

print('The model parameters are, slope (b1) = ' + str(round(linear.slope,2)) + ', and the intercept (b0) = ' + str(round(linear.intercept,2)))
plt.scatter(den, por, color = 'red',edgecolor='black',alpha=0.2,label='sample data')
plt.plot(denv, linear.intercept + linear.slope*denv, 'black', label='linear regression model')
plt.title('Sample Data and Model'); plt.xlabel('Density (g/cm3)'); plt.ylabel('Porosity (%)')
plt.legend(); plt.grid()
plt.show()
```

#### Model Confidence Intervals

Let's calculate the 95% confidence interval for the linear regression model slope parameter, $b_1$, of our model.

* we first need the $t_{critical}$ value, given $alpha = 0.05$ and $df = n-2$.

```python
alpha = 0.05
tstat = st.t.ppf([alpha/2,1-alpha/2], len(por)-2)
slope_lower,slope_upper = linear.slope + tstat*linear.stderr
#intercept_lower,intercept_upper = linear.intercept + tstat*linear.intercept_stderr

print('The model parameters confidence intervals at a ' + str(1-alpha) + ' significance level are:')
print('Slope: ' + str(round(slope_lower,2)) + ' , ' + str(round(slope_upper,2)))
#print('Intercept: ' + str(round(intercept_lower,2)) + ' , ' + str(round(intercept_upper,2)))
```

    The model parameters confidence intervals at a 0.95 significance level are:
    Slope: -10.26 , -7.94

#### Model Prediction Intervals

Let's calculate the prediction intervals.

```python
new_x = 2.05
apha = 0.05

por_hat = linear.intercept + linear.slope*den
MSE = np.sum(np.power(por-por_hat,2))/(len(por)-2)
est_stderr = math.sqrt(1 + 1/len(por) + np.power(new_x - np.average(den),2)/np.sum(np.power(den-np.average(den),2)))
tstat = st.t.ppf([alpha/2,1-alpha/2], len(por)-2)

y_pred_lower, y_pred_upper = linear.intercept + linear.slope*new_x + tstat*math.sqrt(MSE)*est_stderr

plt.scatter(den, por, color = 'red',edgecolor='black',alpha=0.2,label='sample data',zorder=1)
plt.plot(denv, linear.intercept + linear.slope*denv, 'black', label='linear regression model',zorder=1)
plt.scatter(new_x, linear.intercept + linear.slope*new_x, color='yellow',edgecolor='black',label='linear regression model',zorder=2)
plt.plot([new_x,new_x],[y_pred_lower,y_pred_upper],color='black',linestyle='dashed',zorder=1,label='prediction interval')
plt.title('Sample Data, Model and a Prediction Interval'); plt.xlabel('Density (g/cm3)'); plt.ylabel('Porosity (%)')
plt.legend(); plt.grid()
plt.show()

```

#### Model Checking

Let's test the slope with the following hypothesis test:

\begin{equation}
H_0: b_{1} = 0.0
\end{equation}

\begin{equation}
H_1: b_{1} \ne 0.0
\end{equation}

and see if we can reject this hypothesis, $H_{0}$ , that the slope parameter is equal to 0.0.  If we reject this null hypothesis, we show that the slope is meaning full and there is a linear relationship between density and porosity that we can use.

Fortunately, the $linregress$ function from the $stats$ package provides us with the two sided p-value for this test.

```python
print('The linear regression model slope parameter p-value is ' + str(round(linear.pvalue,3)) + '.')
```

    The linear regression model slope parameter p-value is 0.0.

We reject the null hypothesis and adopt the alternative hypothesis, $H_1$, that the slope is not equal to 0.0.

#### Correlation Coefficien and $r^2$ Values

We can also observe correlation coefficient, $\rho^2 = r^2$, and the $r^2$ value that indicates the proportion of variance that is described for our model.

* valid for our linear model

```python
print('The correlation coefficient is = ' + str(round(linear.rvalue,2)) + ' and the r-squared value = ', str(round(linear.rvalue**2,2)))
```

    The correlation coefficient is = -0.84 and the r-squared value =  0.7

#### Model Cross Validation

Let's use this model to make a prediction at all the data locations.

* now plot a standard model cross validation plot, actual vs. predicted values for the response feature
* note, we are only looking at data used to build the model, known as training data.

```python
por_hat = linear.slope * den + linear.intercept
plt.scatter(por_hat,por,color='red',alpha=0.2,edgecolor='black')
plt.plot([0,30],[0,30],color='black',linewidth=1)
plt.ylabel('Actual Porosity (%)'); plt.xlabel('Estimated Porosity (%)'); plt.title('Training Data Cross Validation Plot')
plt.xlim(5,20); plt.ylim(5,20)
plt.grid(); plt.show()
```

Now let's look at the distribution of estimates.

```python
plt.hist(por_hat,color='red',alpha=0.2,edgecolor='black')
plt.title("Porosity Predictions with Linear Model")
plt.xlabel('Porosity (%)'); plt.ylabel('Frequency')
plt.show()
```

It is useful to plot the predictions of porosity and porosity data vs. the density data. From this plot we can observe the linear limitation of our model and get a sense of the unexplained variance $\frac{\sum_{i=1}^{n}(y_i - \hat{y}_i)^2} {n-1}$

```python
plt.scatter(den, por,color='red',alpha=0.2,edgecolor='black',label='sample data')
plt.scatter(den, por_hat,color='blue',alpha=0.2,edgecolor='black',label='model')
plt.title('Sample Data and Model'); plt.xlabel('Density (g/cm3)'); plt.ylabel('Porosity (%)')
plt.legend()
plt.show()
```

Next let's calculate the residual and check their distribution.

* residuals are the true values at the data locations minus the estimates at the data locations, $y_i - \hat{y}_i$.

We want to make sure the average is close to 0.0 (unbiased estimates) and to observe the shape and spread of the residual distribution.

```python
residual = por - por_hat
plt.hist(residual,color='red',alpha=0.2,edgecolor='black')
plt.title("Residual")
plt.xlabel('Porosity True - Estimate (%)')
print('The average of the residuals is ' + str(round(np.mean(residual),2)))
```

Next we will check the residual vs. the fitted value.

* we want to see if the errors are consistent over the range of fitted values.

* for example, we could use this plot to identify higher error or systematic under- or overestimation over a specific range of fitted values.

```python
plt.scatter(por_hat,residual,color='red',alpha=0.2,edgecolor='black')
plt.title('Residual vs. Fitted Value')
plt.xlabel('Porosity Estimate (%)')
plt.ylabel('Porosity Residual (%)')
plt.plot([5,20], [0,0],'black')
plt.show()
```

#### Comments

This was a basic demonstration of linear regression in Python with the SciPy package.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('bmh')
import linearsolve as ls
%matplotlib inline
```

# New-Keynesian Model

This program verifies the math underlying the interactive simulation of the new-Keynesian model: https://www.briancjenkins.com/simulations/nk.html

## Equilibrium Conditions

The equilibrium conditions of the new-Keynesian model:

\begin{align}
y_{t} & = E_ty_{t+1} - \frac{1}{\sigma} r_t + g_t\\
\pi_t & = \beta E_t\pi_{t+1} + \kappa y_t + u_t\\
i_t & = \phi_{\pi} \pi_t + \phi_y y_t + v_t\\
r_t & = i_t - E_t \pi_{t+1}\\
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}\\
v_{t+1} & = \rho_v v_{t} + \epsilon^v_{t+1}
\end{align}

## Analytic Solution

The model's endogenous variables as functions of exogenous state variables:

\begin{align}
y_{t} & = a_1g_t + a_2u_t + a_3v_t\\
\pi_{t} & = b_1g_t + b_2u_t + b_3v_t\\
i_{t} & = c_1g_t + c_2u_t + c_3v_t\\
r_{t} & = d_1g_t + d_2u_t + d_3v_t
\end{align}

where:

\begin{align}
a_1 & = \frac{1-\beta\rho_g}{(1-\beta\rho_g)(1-\rho_g+\sigma^{-1}\phi_y)+ \sigma^{-1}\kappa(\phi_{\pi}-\rho_g)}\\
a_2 & = -\frac{\sigma^{-1}(\phi_{\pi} - \rho_u)}{(1-\beta\rho_u)(1-\rho_u+\sigma^{-1}\phi_y)+ \sigma^{-1}\kappa(\phi_{\pi}-\rho_u)}\\
a_3 & = -\frac{\sigma^{-1}(1-\beta\rho_v)}{(1-\beta\rho_v)(1-\rho_v+\sigma^{-1}\phi_y)+ \sigma^{-1}\kappa(\phi_{\pi}-\rho_v)}
\end{align}

and:

\begin{align}
b_1 & = \frac{\kappa}{(1-\beta\rho_g)(1-\rho_g+\sigma^{-1}\phi_y)+ \sigma^{-1}\kappa(\phi_{\pi}-\rho_g)}\\
b_2 & = \frac{1-\rho_u+\sigma^{-1}\phi_y}{(1-\beta\rho_u)(1-\rho_u+\sigma^{-1}\phi_y)+ \sigma^{-1}\kappa(\phi_{\pi}-\rho_u)}\\
b_3 & = -\frac{\sigma^{-1}\kappa}{(1-\beta\rho_v)(1-\rho_v+\sigma^{-1}\phi_y)+ \sigma^{-1}\kappa(\phi_{\pi}-\rho_v)}\\
\end{align}

and:

\begin{align}
c_1 & = \phi_ya_1 + \phi_{\pi}b_1\\
c_2 & = \phi_ya_2 + \phi_{\pi}b_2\\
c_3 & = 1+ \phi_ya_3 + \phi_{\pi}b_3\\
\end{align}

and:

\begin{align}
d_1 & = c_1 - \rho_g b_1\\
d_2 & = c_2 - \rho_u b_2\\
d_3 & = c_3 - \rho_v b_3\\
\end{align}

## Compute Solution with `linearsolve`

```python
# Input model parameters
beta = np.exp(-2/100)
sigma= 1
kappa= 0.25

phi_pi= 1.5
phi_y = 0.5

rho_g = 0.25
rho_u = 0.35
rho_v = 0.5

parameters=pd.Series()
parameters.beta=beta
parameters.sigma=sigma
parameters.kappa=kappa
parameters.phi_pi=phi_pi
parameters.phi_y=phi_y
parameters.rho_g=rho_g
parameters.rho_u=rho_u
parameters.rho_v=rho_v

def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters
    p = parameters

    # Variables
    fwd = variables_forward
    cur = variables_current

    # Euler equation
    euler_eqn = fwd.y -1/p.sigma*cur.r + cur.g - cur.y

    # NK Phillips curve
    phillips_curve = p.beta*fwd.pi + p.kappa*cur.y + cur.u - cur.pi

    # Interest rate rule for monetary policy
    interest_rule = p.phi_y*cur.y+p.phi_pi*cur.pi + cur.v - cur.i

    # Fisher equation
    fisher_eqn = cur.i - fwd.pi - cur.r

    # Exogenous demand
    g_proc =  p.rho_g*cur.g - fwd.g

    # Exogenous inflation
    u_proc =  p.rho_u*cur.u - fwd.u

    # Exogenous monetary policy
    v_proc =  p.rho_v*cur.v - fwd.v

    # Stack equilibrium conditions into a numpy array
    return np.array([
            euler_eqn,
            phillips_curve,
            interest_rule,
            fisher_eqn,
            g_proc,
            u_proc,
            v_proc
        ])

# Initialize the nk
nk = ls.model(equilibrium_equations,
              nstates=3,
              varNames=['g','u','v','i','r','y','pi'],
              parameters=parameters)

# Set the steady state of the nk
nk.set_ss([0,0,0,0,0,0,0])

# Find the log-linear approximation around the non-stochastic steady state
nk.approximate_and_solve(loglinear=False)

# Solve the nk
nk.solve_klein(nk.a,nk.b)
```

## Compute Solution Directly

```python
a1=(1-beta*rho_g)/((1-beta*rho_g)*(1-rho_g+phi_y/sigma)+kappa/sigma*(phi_pi-rho_g))
a2=-(phi_pi-rho_u)/sigma/((1-beta*rho_u)*(1-rho_u+phi_y/sigma)+kappa/sigma*(phi_pi-rho_u))
a3=-(1-beta*rho_v)/sigma/((1-beta*rho_v)*(1-rho_v+phi_y/sigma)+kappa/sigma*(phi_pi-rho_v))

b1=kappa/((1-beta*rho_g)*(1-rho_g+phi_y/sigma)+kappa/sigma*(phi_pi-rho_g))
b2=(1-rho_u+phi_y/sigma)/((1-beta*rho_u)*(1-rho_u+phi_y/sigma)+kappa/sigma*(phi_pi-rho_u))
b3=-kappa/sigma/((1-beta*rho_v)*(1-rho_v+phi_y/sigma)+kappa/sigma*(phi_pi-rho_v))

c1=phi_y*a1+phi_pi*b1
c2=phi_y*a2+phi_pi*b2
c3=phi_y*a3+phi_pi*b3+1

d1=c1-rho_g*b1
d2=c2-rho_u*b2
d3=c3-rho_v*b3
```

## Compare Analytic and Numeric Solutions

```python
print('verify a1,a2, a3:')
print(a1-nk.f[2,0])
print(a2-nk.f[2,1])
print(a3-nk.f[2,2])

print('\n')
print('verify b1,b2, b3:')
print(b1-nk.f[3,0])
print(b2-nk.f[3,1])
print(b3-nk.f[3,2])

print('\n')
print('verify c1,c2, c3:')
print(c1-nk.f[0,0])
print(c2-nk.f[0,1])
print(c3-nk.f[0,2])

print('\n')
print('verify d1,d2, d3:')
print(d1-nk.f[1,0])
print(d2-nk.f[1,1])
print(d3-nk.f[1,2])
```

    verify a1,a2, a3:
    1.3234889800848443e-23
    -3.970466940254533e-23
    -1.933735852167675e-17

    verify b1,b2, b3:
    1.1102230246251565e-16
    -2.6407226218542156e-16
    1.7049828215931484e-10

    verify c1,c2, c3:
    2.220446049250313e-16
    -3.9610839327813233e-16
    2.557477007947284e-10

    verify d1,d2, d3:
    4.440892098500626e-16
    -1.6344237900633632e-16
    1.0070241020212785e-10

## Plot Simulations

```python
# Compute impulse responses and plot
nk.impulse(T=15,t0=1,shocks=[0.2,0.5,1])

# Create the figure and axes
fig = plt.figure(figsize=(12,12))
ax1 = fig.add_subplot(3,1,1)
ax2 = fig.add_subplot(3,1,2)
ax3 = fig.add_subplot(3,1,3)

# Plot commands
nk.irs['e_g'][['y','pi','r']].plot(lw='5',alpha=0.5,grid=True,title='Demand shock',ax=ax1).legend(loc='upper right',ncol=5)
nk.irs['e_u'][['y','pi','r']].plot(lw='5',alpha=0.5,grid=True,title='Inflation shock',ax=ax2).legend(loc='upper right',ncol=5)
nk.irs['e_v'][['y','pi','r']].plot(lw='5',alpha=0.5,grid=True,title='Interest rate shock',ax=ax3).legend(loc='upper right',ncol=5)

fig.tight_layout()

```

```python

```


```python
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import time as t

%matplotlib inline
```

```python
# Setting the threshold, where to stop the calculation
threshold = 0.1
```

### Loading the .npy files

```python
masses = np.load('files/masses.npy')
positions = np.load('files/positions.npy')
velocities = np.load('files/velocities.npy')

print(positions.shape)
```

    (100, 2)

```python
#use s to increase or reduce
s = len(positions)

position_x = positions[:,0][0:s]
position_y = positions[:,-1][0:s]
initial_velocity_x = velocities[:,0][0:s]
initial_velocity_y = velocities[:,-1][0:s]
acc_x = np.zeros(len(position_x))[0:s]
acc_y = np.zeros(len(position_y))[0:s]
mass = masses.reshape(len(masses),)[0:s]
```

### Defining constants and placeholders required

```python
t2_cons = tf.constant(0.5,dtype=tf.float64,name='cons_fac')
time  = tf.constant(0.0001,dtype=tf.float64,name="time_step")
G = tf.constant(6.67*pow(10,5),dtype=tf.float64,name="G-constant")
neg = tf.constant(-1.0,dtype=tf.float64)
```

```python
px = tf.placeholder(tf.float64,name="px")
py = tf.placeholder(tf.float64,name="py")
ux = tf.placeholder(tf.float64,name="ux")
uy = tf.placeholder(tf.float64,name="uy")
ax = tf.placeholder(tf.float64,name="ax")
ay = tf.placeholder(tf.float64,name="ay")
m =  tf.placeholder(tf.float64,name="mass")
```

#### Calculating initial euclidean distance and cube of distance for acceleration using broadcasting for outer subtraction

```python
#for x
dis_x = tf.transpose([px]) - px

#for y
dis_y = tf.transpose([py]) - py

dis_z = tf.sqrt(tf.add(tf.square(dis_x),tf.square(dis_y)))

cube_dis_z = tf.pow(dis_z,3)
```

#### Calculating acceleration in x and y direction for each particle due to graviational force

```python
#For ax
divide_mass_mul_x = tf.divide(tf.multiply(m,dis_x),cube_dis_z)

#removing nan, where cube_dis_z is zero
divide_mass_mul_x = tf.where(tf.is_nan(divide_mass_mul_x), tf.zeros_like(divide_mass_mul_x), divide_mass_mul_x)

#summation of all acceleration and multiplied with G(graiational constant)
sum_mass_mul_x = tf.multiply(G,tf.reduce_sum(divide_mass_mul_x,1))

#Final acceleration along x direction
final_acc_x = tf.multiply(neg,sum_mass_mul_x)

########################################################

#For ay
divide_mass_mul_y = tf.divide(tf.multiply(m,dis_y),cube_dis_z)

#removing nan, where cube_dis_z is zero
divide_mass_mul_y = tf.where(tf.is_nan(divide_mass_mul_y), tf.zeros_like(divide_mass_mul_y), divide_mass_mul_y);

#summation of all acceleration and multiplied with G(graiational constant)
sum_mass_mul_y = tf.multiply(G,tf.reduce_sum(divide_mass_mul_y,1))

# Final acceleration along x direction
final_acc_y = tf.multiply(neg,sum_mass_mul_y)
```

#### Updating position of particle along x and y direction according to eqn

\begin{align}
\ x & = \ x_0 + ut+0.5at^2  \\
\end{align}

```python
## For x position
t1_x = tf.multiply(ux,time,name="t1_x")
t2_x = tf.multiply(tf.multiply(t2_cons,final_acc_x),tf.square(time))

tx = tf.add(px,tf.add(t1_x,t2_x))
```

```python
# For y position
t1_y = tf.multiply(uy,time,name="t1_y")
t2_y = tf.multiply(tf.multiply(t2_cons,final_acc_y),tf.square(time))

ty = tf.add(py,tf.add(t1_y,t2_y))
```

#### Updating velocity after one time step

\begin{align}
\ v & = \ u + at  \\
\end{align}

```python
# for x direction
v_x = tf.add(ux,tf.multiply(final_acc_x,time))

# for y direction
v_y = tf.add(uy,tf.multiply(final_acc_y,time))
```

#### Computation of distance matrix again to get minimum distance and check for threshold

```python
#for x
dis_x_new = tf.transpose([tx]) - tx

#for y
dis_y_new = tf.transpose([ty]) - ty

dis_z_new = tf.sqrt(tf.add(tf.square(dis_x_new),tf.square(dis_y_new)))

max_dis_z_new = tf.reduce_max(dis_z_new)

#change the diagonal matrix to find min element and replacing with max matrix element.Because diagonal is zero
dis_z_new = tf.matrix_set_diag(dis_z_new,s*[max_dis_z_new])

min_dis_z_new = tf.reduce_min(dis_z_new)
```

#### Initial values in placeholders

```python
feed_dict = {
    px:position_x,
    py:position_y,
    ux:initial_velocity_x,
    uy:initial_velocity_y,
    ax:acc_x,
    ay:acc_y,
    m:mass
}
```

```python
def plot(x,y,counter):
    plt.ylim(-10, 80)
    plt.xlim(-10, 80)

    #plt.scatter(x, y)
    plt.scatter(x[8], y[8],c='red',s=50)
    plt.scatter(x[91], y[91],c='blue',s=50)
    plt.savefig('As1Q2_vid/'+str(counter)+'.png')
    plt.close()
```

```python
start = t.time()

count_iter = 0

with tf.Session() as session:

    while(True):
        result = session.run([tx,ty,v_x,v_y,final_acc_x,final_acc_y,min_dis_z_new],feed_dict=feed_dict)

        # Writing graph summary in logs
        #writer = tf.summary.FileWriter("./logs/graph",session.graph)

        ##Updating the values
        feed_dict[px]=result[0]
        feed_dict[py]=result[1]
        feed_dict[ux]=result[2]
        feed_dict[uy]=result[3]
        feed_dict[ax]=result[4]
        feed_dict[ay]=result[5]

        #Save plot for the video
        #plot(result[0],result[1],count_iter)

        ### checking if minimum distance between any two points is below threshold
        if result[6]<=threshold:
            break

        count_iter+=1

end = t.time()
print("Time Taken: ",end - start)
```

    Time Taken:  2.1669862270355225

#### Save position and velocity as .npy file

```python
final_pos = []
final_vel = []

for i in range(len(result[0])):
    final_pos.append([result[0][i],result[1][i]])
    final_vel.append([result[2][i],result[3][i]])
```

```python
final_pos = np.array(final_pos,dtype='float64')
final_vel = np.array(final_vel,dtype='float64')
```

```python
np.save('output/positions.npy', final_pos)
np.save('output/velocities.npy', final_vel)
```

## Tensorbaord Visualization
- tensorboard --logdir=path/to/log-directory

## Video Creation

```python
import os
import cv2
```

```python
f=os.listdir("As1Q2_vid")

sorted_files = sorted(f,key=lambda x:int(x.split('.')[0]))

img_frame = []

for file in sorted_files:
    path="As1Q2_vid/"
    img = cv2.imread(path+file)
    h,w,l = img.shape
    size=(w,h)
    img_frame.append(img)

out = cv2.VideoWriter("As1Q2.mp4",0x7634706d,10,size)
```

```python
for i in range(len(img_frame)):
    out.write(img_frame[i])
out.release()
```

```python

```

```python

```


# Further information

## How can we plot a function

It is possible to plot a function using `sympy` using the `sympy.plot` function:

````{tip}
```
sympy.plot(expression)
```
````

So for example, here is a plot of $f(x)=x^2 + 3x + 1$:

```python
import sympy as sym

x = sym.Symbol("x")
sym.plot(x ** 2 + 3 * x + 1)
```

It is possible to specify the x limits and combine it with other plots:

```python
sym.plot(x ** 2 + 3 * x + 1, xlim=(-5, 5))
```

**This plotting solution is fine it you want to take a look at a function
quickly but it is not recommended.** The main library for plotting is called
`matplotlib` and {ref}`matplotlib` covers this.

- Here is the `sympy` documentation for plotting:
  <https://docs.sympy.org/latest/modules/plotting.html>
- Here is the official `matplotlib` documentation: <https://matplotlib.org>