# Text Message Spam Detection

## The Business Use Case

You are the CEO of a new email service company trying to attract capital for your next growth stage. Most of the private equity firms you have spoken to want to see a user base of at least 100,000 users prior to commiting. Due to your superb marketing team, you estimate that each day you attract 1000 new users. Typically, 10% of all email messages are spam, and an average user receives 50 emails per day.

One of your data scientists says he can provide you a model with **90% accuracy** in classifying spam / ham. Another data scientist says she can build a model that is only **80% accurate** but has **100% recall**. A third data scientist says he can build a model that has **80% accuracy** with **100% precision**.

Which of the above models would you pick to model? Why?

```python
import pandas as pd
data = pd.read_csv("spam-sms.csv", encoding='latin-1')
data.shape
```

    (1037, 2)

```python
Y = data["class"].values
X = data["text"].values
```

# Bayes Rule

$$
\begin{equation}
P(A|B) = \frac{P(B|A)P(A)}{P(B)}
\end{equation}
$$

For our purposes, we will redefine this as

$$
\begin{equation}
P(spam|text) = \frac{P(text|spam)P(spam)}{P(text)}
\end{equation}
$$

Here,

- **prior** means before seeing any new text (evidence). Our impression of the likelihood of certain words appearing before new evidence is introduced.
- **text** is the new message (evidence) being introduced that we want to classify as either spam or ham. Let's say we have a new text message `When are you coming home? I'm hungry.`.

- $P(text)$ is the **prior likelihood** of seeing a particular text message with that exact combination of words. For instance, `P("the car is")` will be significantly higher than `P("Downstream supply chain agents")`, especially in the **context of text messages**.
- $P(spam)$ is the likelihood that any text message will be spam. This is computed in our dataset:
```python
p_spam = sum(data["class"] == "spam") / len(data)
p_ham = 1 - p_spam # since there are only two classes
```
- $P(text|spam)$ is our **likelihood**. More specifically, the likelihood of this text message given that it is a piece of spam. It is saying, *let's assume that this message is spam. Knowing that, how likely is it that we'll find this particular combination of words in the text message?*

In order to quickly get our likelihoods, we'll need to create a **likelihood table**:

```python
spam_data = data[data["class"] == "spam"]
ham_data = data[data["class"] == "ham"]
print(f"The shape of the spam_data is {spam_data.shape}. The shape of the ham_data is {ham_data.shape}.")
```

    The shape of the spam_data is (156, 2). The shape of the ham_data is (881, 2).

# Perform Count Vectorization

```python
from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer()
vectorizer.fit(data["text"].values)

# create the vocabulary list
vocabulary = set(vectorizer.get_feature_names())
```

# Populate Likelihood Table

```python
# Create the vocabulary list
import spacy, string
nlp = spacy.load('en') # python3 -m spacy download en
from nltk import word_tokenize

likelihood_table = pd.DataFrame(columns=["spam", "ham"], index=list(vocabulary)).fillna(0)

# populate the spam column in our likelihood table
for i, sentence in enumerate(spam_data["text"].values):
    for token in word_tokenize(sentence):
        if token.lower() not in likelihood_table.index:
            likelihood_table.loc[token.lower(), "spam"] = 1
        else:
            likelihood_table.loc[token.lower(), "spam"] += 1
likelihood_table.fillna(0, inplace=True)
```

```python
# populate the ham column in our likelihood table
for i, sentence in enumerate(ham_data["text"].values):
    for token in nlp(sentence):
        if token.text.lower() not in likelihood_table.index:
            likelihood_table.loc[token.text.lower(), "ham"] = 1
        else:
            likelihood_table.loc[token.text.lower(), "ham"] += 1
likelihood_table.fillna(0, inplace=True)
```

```python
likelihood_table.head()
```

      spam
      ham

      stylish
      0.0
      2.0

      lager
      0.0
      1.0

      been
      8.0
      24.0

      indian
      0.0
      1.0

      child
      0.0
      1.0

### Likelihood with "Fun"

```python
likelihood_table.loc["fun"]
```

    spam    1.0
    ham     2.0
    Name: fun, dtype: float64

### Frequency with "Won"

What about words like `won`?
```python
likelihood_table.loc["won"]
```
Output:
```
spam    16.0
ham      0.0
Name: won, dtype: float64
```
# Edge Cases
What is $P(w = won|c = ham)$? If even one of the words' class-conditional probabilities is 0, then the entire likelihood will be zero, since the likelihood is simply the product of all the words' individual likelihoods.

## Additive Smoothing Techniques

We can define a new likelihood for the word `won`:

$$
\begin{equation}
P_{new}(w = won | c = ham) = \frac{N_{ham, won} + \alpha}{N_{ham} + \alpha d}
\end{equation}
$$

Here, $N_{ham, won}$ is the number of times `won` appears in a text message that is classified as ham, and $N_{ham}$ is simply the total number of messages that are classified as ham.

## Simple Optimizations to Improve Naive Bayes Probabilistic Models for Text Classification

- to may be useful to simply create a simple **co-occurence matrix**, and **run a correlation analysis** on the features (words). If certain words have extremely high correlations, you may wish to take them out, or fuse them into a single entity.
- apply smoothing techniques to handle **out-of-vocabulary test words**
- **ensemble techniques like bagging / boosting** do **not** help. There isn't any "variation" in a Naive Bayes model. Given the same trained corpus $C$, and a new text message $m$, a Naive Bayes model will always output the same prediction.

# Representing Words as Probabilities

We can represent a sentence (a sequence of words) mathematically as

$$
\begin{equation}
w = \{{w_0, w_1, w_2, \dots,w_{s-1}}\}
\end{equation}
$$

Here, **$s$** represents the total number of words in the sentence. **$w_{0}$** represents the first word in the sentence, **$w_{1}$** represents the second word in the sentence, and so on.

# Exercise:

`Older people, like everyone else, can benefit from accessing ride-sharing, but many are not comfortable with smart-phones.`

You can ignore punctuation and capitalization for now.

1. What is $s$?
2. What is $w_4$? What is $w_6$?
3. What is $V$ (this corpus' vocabulary size, assuming this is the only sentence in the corpus)? You can do this the hard way, by counting manually.

```python
sentence = "Older people, like everyone else, can benefit from accessing ride-sharing, but many are not comfortable with smart-phones"

import re # the most efficient, concise way (less readable)
vocabulary = set([re.sub(r'[^\w\s]','',word).lower() for word in sentence.split()])
print("The size of the vocabulary is {} words".format(vocabulary))
```

# Independence

In statistics, two events are independent if the outcome of one event does not affect the probability of the outcomes of another event.

You will also often see this written as
$$
\begin{equation}
P(A,B) = P(A) * P(B)
\end{equation}
$$

In other words, an event A is independent of event B if the **probability of event A and event B happening together** is equal to **the probability of event A multplied by the probability of event B**.

# Bigram Model

A bigram is a group of two tokens (frequently words) that are treated as one distinct entity. For instance, the distinct bigrams in the sentence `I am home now` would be
```python
bigrams = [
    ("I", "am"),
    ("am", "home"),
    ("home", "now")
]
```

### Exercise:
Write a Python function to find all the bigrams in the sentence
`In recent years, Johnson & Johnson has been focusing more on its high-margin pharmaceutical segment via acquisitions.`

**Hints**:
- split the sentence into a list of individual words (`my_sentence.split()`)
- remove punctuation
- lowercase all the letters
- use a for loop to iterate through this list, getting the **i-th** and **i + 1-th** elements of the list

**Challenge**:
Generalize this function to work with `n-grams`.

## Language Model

Are words in a sentence conditionally independent from each other? In other words, does knowing that the first word `The` change your belief in the likelihood of the second word that follows?

Which of the following sentences is more likely?

```python
sentence_A = "Jack went to Wal-Mart."
sentence_B = "at and the be of I"
```
Notice that all the words in sentence B come from [Wikipedia most common words in the English language](https://en.wikipedia.org/wiki/Most_common_words_in_English). Yet we intuitively know that the sentence is nonsensical and is unlikely to be seen in natural language.

We can express the likelihood of a sentence $w$ as $p(w)$, and define it as
$$
\begin{equation}
p(w) = \prod_{i=0}^{s}p(w_{i+1}|w_{i})
\end{equation}
$$

If we want to generalize this to an **N-Gram** model:

$$
\begin{equation}
p(w) = \prod_{i=0}^{s}p(w_i |w_{i-n+1}, w_{i-n+2}, \dots, w_{i})
\end{equation}
$$
$$
\begin{equation}
p(w) = \prod_{i=0}^{s}p(w_i | w_{i-n}^{i})
\end{equation}
$$

Here, $w_{i-n}^{i} = w_{i-n+1}, w_{i-n+2}, \dots, w_{i}$.

### Exercises

Use the following documents:

```python
documents = [
    "Eat dinner at home",
    "He needs to go to the store",
    "She needs to go home"
]
```

1. Get a list of all unique tokens in the vocabulary

2. Calculate the transition frequencies.

3. Calculate the transition probabilities.

## Generating Bigrams Using NLTK

```python
import pandas as pd
import nltk
from nltk import word_tokenize
reviews_df = pd.read_csv("mcdonalds-yelp-negative-reviews.csv", encoding="latin-1")
for review in reviews_df["review"]:
    bigram = list(nltk.bigrams(word_tokenize(review)))
print(bigram[:10])
```

    [('I', 'wanted'), ('wanted', 'to'), ('to', 'grab'), ('grab', 'breakfast'), ('breakfast', 'one'), ('one', 'morning'), ('morning', 'before'), ('before', 'work'), ('work', 'since'), ('since', 'it')]

## Generating Bigrams Using Scikit-Learn

```python
from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(ngram_range=(2,2))
X = vectorizer.fit_transform(reviews_df["review"])

bigram_features = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())
bigram_features.shape
```

    (1525, 64297)

```python
vectorizer = CountVectorizer(ngram_range=(1,1))
X = vectorizer.fit_transform(reviews_df["review"])

unigram_features = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())
unigram_features.shape
```

    (1525, 8379)

## Model Evaluation: Choosing n in an n-Gram Model

- the larger the dataset, and by implication, the more rich the corpus, the larger the n we can likely try.
- in practice, $n = 2$, $n = 3$, $n = 4$ work well. A larger $n$ tends to begin to overfit (and may be computationally extremely expensive). Remember the **bias-variance** tradeoff:

Here, as $n \rightarrow \infty$, model complexity increases dramatically.
- **tune $n$ based on the performance of the downstream model**: usually n-gram models are the first step in a broader sentiment analysis prediction model, or topic modelling model, recommendation system, or sequence-to-sequence translation task.

### Perplexity

Look again at the definition of likelihood for a particular sentence:

$$
\begin{equation}
L = p(w) = \prod_{i=0}^{s}p(w_i | w_{i-n}^{i})
\end{equation}
$$

Here, $w_{i-n}^{i}$ stands for $w_{i-n}, w_{i-n+1}, ..., w_{i-1}$ (if you have bi-gram (**`n=2`**) model, then you would have $w_{i-2}, w_{i-1}$)

Is it reasonable to compare two sentences, one with $s=4$ (sentence length of 4 words) with the one below in Sentence B?

##### Sentence A:
> *I love to eat.*

##### Sentence B:
> *My escort was an exceptionally genial sixty-seven-year-old man named Don Seely, an electrical engineer who said that he was between jobs and using the unwanted free time to volunteer his services to the Northern Kentucky Tea Party, the rally’s host organization, as a Webmaster.*

Answer: **No**. A common way of quantifying the likelihood of your n-gram models, accounting for different sizes of test corpuses, is to use **perplexity**. Remember that our likelihood of seeing a particular sentence is

$$
\begin{equation}
P = \frac{1}{\sqrt[N]{p(w)}}
\end{equation}
$$
$N$ is the length of all the words in the test sentence. We typically use perplexity, instead of simply likelihood, as the overall model evaluation metric, because in general, **in order to compare two different models**, they should be using the same test corpus / vocabulary.

```python
import numpy as np
import matplotlib.pyplot as plt

def perplexity(likelihood, N):
    return 1 / (likelihood ** (1/N))

x = []
y = []
for i in range(5, 500):
    x.append(i)
    y.append(perplexity(likelihood=.204, N=i)) # an example likelihood of .004

plt.plot(x, y)
plt.xlabel("N")
plt.ylabel("Perplexity")
```

## Dealing with Out-of-Vocabulary Words

Let's pretend that our training corpus is

> *This is mistaken logic. It is true that a high variance and low bias model can preform well in some sense.*

Our test corpus is

> *This **is not** true.*

Assuming a bi-gram model is used, what is the **perplexity** of our model?

We don't actually need to count each bi-gram. **The answer is infinity**. Why?
$
\begin{equation}
p(w_i = not | w_{i-1} = is) = 0
\end{equation}
$

What you can do instead:

* Look at the **frequency distribution** of words in your corpus
* Decide upon some **threshold cutoff**, where every word below that threshold frequency will be converted into an **`UNKNOWN`** token. Now, whenever a new word appears that is out of vocabulary, you simply convert it into **`UNKNOWN`** and run the tests as usual.

week3/Probabilities and N-Gram Language Models.ipynb


```python
%pylab inline
rcParams['savefig.dpi']=100
```

    Populating the interactive namespace from numpy and matplotlib

```python
from scipy import  stats
```

```python
chisq=stats.chi2(1)
```

```python
fig,ax=subplots()
xi = np.linspace(0.01,10,100)
ax.plot(xi,chisq.pdf(xi),color='k',lw=2)
ax.axis(ymax=1)
ax.set_xlabel('$x$',fontsize=18)
ax.set_ylabel('$f_x(x)$',fontsize=22)
ax.set_title(r'$\chi_1^2$ Probability Density',fontsize=18,fontdict={'family':'serif'})
fig.savefig('../fig-probability/ProbabilityInequalities_001.png')
```

```python
fig,ax=subplots()
ax.plot(xi,1-chisq.cdf(xi),label=r'$\mathbb{P}(X>\epsilon)$',color='k',linestyle='-',lw=3)
ax.plot(xi,1/xi,label=r'$\mathbb{E}(X)/\epsilon$',lw=3,color='k',ls='--')
ax.fill_between(xi,1-chisq.cdf(xi),1/xi,color='gray',alpha=.3)
ax.axis(ymax=.5)
ax.set_xlabel('$\epsilon$',fontsize=18)
ax.legend(fontsize=18)
ax.set_title("Markov's Inequality",fontsize=18)
fig.savefig('../fig-probability/ProbabilityInequalities_002.png')
```

```python
chisq.mean()
```

    1.0

```python
from scipy.integrate import quad
rv = stats.chi2(1)
```

```python
# from sympy import init_printing;init_printing()
```

```python
import sympy
from sympy import stats as ss
t = sympy.symbols('t',real=True,positive=True)
# z = sympy.symbols('z',real=True,positive=True)

x=ss.ChiSquared('x',1)
```

```python
# v=ss.P((x-1) > t,x>1)+ss.P(-(x-1) > t,x<1)
```

```python
w=(1-ss.cdf(x)(t+1))+ ss.cdf(x)(1-t)
```

```python
fig,ax=subplots()
fw=sympy.lambdify(t,w)
ti = np.linspace(.01,5)
fvals = map(fw,ti)
ax.plot(ti,fvals,'k',lw=3,label=r'$\mathbb{P}(\vert X-\mu\vert > \epsilon)$')
ax.plot(ti,4/ti**2,'k--',lw=3,label=r'$\sigma^2/\epsilon^2$')
ax.fill_between(ti,fvals,4/ti**2,color='gray',alpha=.3)
ax.axis(ymax=1)
ax.set_xlabel('$\epsilon$',fontsize=18)
ax.set_title("Chebyshev's Inequality")
ax.legend(fontsize=16)
fig.savefig('../fig-probability/ProbabilityInequalities_003.png',dpi=100)
```

```python
%qtconsole
```

```python
# from sympy.interactive import printing
# printing.init_printing(use_latex=True)
```

```python
from IPython.display import Math
Math(sympy.latex(w))
```

$$\begin{cases} \operatorname{erf}{\left (\frac{\sqrt{2}}{2} \sqrt{- t + 1} \right )} & \text{for}\: - t + 1 \geq 0 \\0 & \text{otherwise} \end{cases} - \begin{cases} \operatorname{erf}{\left (\frac{\sqrt{2}}{2} \sqrt{t + 1} \right )} & \text{for}\: t + 1 \geq 0 \\0 & \text{otherwise} \end{cases} + 1$$

```python
Math(sympy.latex(ss.cdf(x)(t)))
```

$$\begin{cases} \operatorname{erf}{\left (\frac{\sqrt{2} \sqrt{t}}{2} \right )} & \text{for}\: t \geq 0 \\0 & \text{otherwise} \end{cases}$$

## Hoeffding’s Inequality

```python
from  scipy.integrate import quad
from scipy.interpolate import  interp1d

def pdf(n=3,m=20):
    u = np.ones(m)
    fv=reduce(lambda i,j:np.convolve(i,j),[u,]*n,u)
    xv = np.linspace(0,n,len(fv))
    norm = (xv[1]-xv[0])*sum(fv)
    xv = xv-np.mean(xv)
    return interp1d(xv,fv/norm,fill_value=0,bounds_error=False)

f3=pdf(5)
apdf=interp1d(f3.x[f3.x>=0],f3.y[f3.x>=0]*2,fill_value=0,bounds_error=False)

ti = linspace(0.01,f3.x.max(),50)

plot(f3.x,f3.y,'-')
```

```python
fig,ax=subplots()

vals = map(lambda t:quad(apdf,t,100)[0],ti)
ax.plot(ti,vals,'-k',label=r'$\mathbb{P}(\vert \overline{X}_n-1/2\vert \geq \epsilon)$',lw=3)
ax.plot(ti,2*np.exp(-2*ti**2),label='Hoeffding',color='k',ls='--',lw=3)
ax.fill_between(ti,vals,2*np.exp(-2*ti**2),color='gray',alpha=0.3)
ax.plot(ti,0.52/ti,label='Markov',color='k',ls=':')
ax.legend(loc=0,fontsize=12)
ax.axis(ymax=2)
ax.set_xlabel('$\epsilon$',fontsize=18)
fig.savefig('../fig-probability/ProbabilityInequalities_004.png',dpi=100)
```

```python

```


# Nonlinear regression for inverse dynamics

In this question, we fit a model which can predict what torques a robot needs to apply in order to make its arm reach a desired point in space. The data was collected from a SARCOS robot arm with $7$ degrees of freedom. The input vector $x \in \mathbb{R}^{21}$ encodes the desired position, velocity and accelaration of the $7$ joints. The output vector $y \in \mathbb{R}^7$ encodes the torques that should be applied to the joints to reach that point. The mapping from $x$ to $y$ is highly nonlinear.

We can find the data at [http://www.gaussianprocess.org/gpml/data/](http://www.gaussianprocess.org/gpml/data/). Obviously the first step is to read this data.

```python
%matplotlib inline

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import time
from util import read_all_sarcos_data, TARGET_COLUMN

training_data, test_data = read_all_sarcos_data()
training_data.head()
```

      x1
      x2
      x3
      x4
      x5
      x6
      x7
      x8
      x9
      x10
      ...
      x19
      x20
      x21
      y1
      y2
      y3
      y4
      y5
      y6
      y7

      0
      0.019478
      -0.134218
      0.027439
      1.516401
      0.300936
      0.058259
      0.150134
      -0.266791
      -0.237134
      -0.091272
      ...
      11.695956
      1.210212
      -22.119289
      50.292652
      -36.971897
      20.937170
      47.821712
      -0.424812
      -0.907553
      8.090739

      1
      0.017279
      -0.137077
      0.026999
      1.532517
      0.301344
      0.058259
      0.128653
      -0.153640
      -0.335279
      0.006449
      ...
      14.643369
      1.015070
      -17.048688
      44.104164
      -28.851845
      16.230194
      43.194073
      -0.228739
      -1.235817
      7.762475

      2
      0.016336
      -0.140878
      0.027250
      1.549670
      0.302318
      0.059027
      0.104104
      -0.047313
      -0.418732
      0.106274
      ...
      15.467628
      0.910548
      -11.415526
      37.354858
      -20.809343
      12.379975
      39.386017
      0.244491
      -1.700880
      7.289678

      3
      0.016273
      -0.145307
      0.029072
      1.566855
      0.307628
      0.059027
      0.080321
      0.053238
      -0.460963
      0.188013
      ...
      10.309203
      0.921360
      -5.772058
      30.676065
      -13.963816
      7.702940
      36.478813
      -0.182062
      -2.143370
      6.410800

      4
      0.017279
      -0.150051
      0.031083
      1.584416
      0.314162
      0.059027
      0.058840
      0.133810
      -0.462264
      0.263975
      ...
      2.868096
      1.059957
      -0.491542
      25.920128
      -11.178479
      5.643934
      34.773911
      -1.031687
      -2.355776
      5.792892

5 rows × 28 columns

Let's standardize our data by making each column have mean $0$ and unit variance.

## Linear Regression

Now, the first order of business is to fit a standard linear regression. I'll just use the `sklearn` implementation.

```python
from sklearn import preprocessing
def standardize_data(data):
    feature_column_names = [column_name for column_name in all_data.columns if 'x' in column_name]
    features = data[feature_column_names]
    # only care about the first output for this problem
    target = data[TARGET_COLUMN]
    X = features.as_matrix()
    Y = target.as_matrix()
    X_scaled = preprocessing.scale(X, axis = 0, with_mean = True, with_std = True, copy = True)
    # only standardize inputs
    # Y_scaled = preprocessing.scale(Y, axis = 0, with_mean = True, with_std = True, copy = True)
    return X_scaled, Y
# standardize together and separate
all_data = pd.concat([training_data, test_data], axis = 0)
standardize_data(all_data)
X, Y = standardize_data(all_data)
X_scaled, Y_scaled = X[:len(training_data)], Y[:len(training_data)]
X_scaled_test, Y_scaled_test = X[len(training_data):], Y[len(training_data):]
```

```python
from sklearn import linear_model
## since data is already normalized
linearRegressor = linear_model.LinearRegression(fit_intercept = True, normalize = False)
linearRegressor.fit(X_scaled, Y_scaled)
linearRegressor.score(X_scaled_test, Y_scaled_test)
```

    /usr/local/lib/python3.6/site-packages/scipy/linalg/basic.py:1018: RuntimeWarning: internal gelsd driver lwork query error, required iwork dimension not returned. This is likely the result of LAPACK bug 0038, fixed in LAPACK 3.2.2 (released July 21, 2010). Falling back to 'gelss' driver.
      warnings.warn(mesg, RuntimeWarning)

    0.92494581827142286

According to the problem, we should be able to get a standardized mean squared error (SMSE) of 0.075. We define

\begin{equation}
SMSE = \left(\frac{1}{N_{test}}\sum_{i=1}^{N_{test}}\left(y_i - \hat{y}_i\right)^2\right)/\sigma^2,
\end{equation}

where

\begin{equation}
\sigma^2 = \frac{1}{N_{train}}\sum_{i=1}^{N_{train}}\left(y_i - \bar{y}\right)^2.
\end{equation}

```python
def compute_mse(prediction, labels):
    return (sum((labels - prediction)**2)/len(prediction))
def compute_model_mse(model, X, y):
    predicted_y = model.predict(X)
    return compute_mse(predicted_y, y)
def log_mse(mse):
    print('MSE: {:.10f}'.format(mse))
    print('SMSE: {:.10f}'.format(mse/np.var(Y_scaled, ddof = 0)))

test_mse = compute_model_mse(linearRegressor, X_scaled_test, Y_scaled_test)

log_mse(test_mse)
```

    MSE: 31.0857592861
    SMSE: 0.0742260930

Looks like we got the correct result. Yay!

## Radial Basis Function Network

Next, we'll try using a radial basis function (RBF) network. First, we need to do some K-means clustering. We'll pick the number of clusters with the qualitative kink, knee, or elbow method (three different names for the same method). The book says to use cross-validation, but that's nonsense since the book itself says that using cross-validation is not possible for non-probablistic methods. I guess we could use a Gaussian Mixture Model if we want to do cross-validation?

We'll fit the model with various values of $k$ and look at the reconstruction error.

```python
from sklearn import cluster
from joblib import Parallel, delayed

k_range = np.arange(1, 107, 3)
def score_k(k):
    kMeansModel = cluster.KMeans(n_clusters = k, init = 'k-means++', algorithm = 'elkan', precompute_distances = True)
    kMeansModel.fit(X_scaled)
    return -kMeansModel.score(X_scaled_test)
start = time.perf_counter() ## time how long it takes
reconstruction_error = Parallel(n_jobs = 2, backend='threading')(delayed(score_k)(k) for k in k_range)
print('Time elapsed: ' + str(time.perf_counter() - start))
reconstruction_error
```

    Time elapsed: 963.612736572999

    [93511.741947211151,
     65971.803182832591,
     52906.722708804191,
     45695.425134108205,
     40914.984232328658,
     37248.225253985787,
     34558.310657963739,
     32687.881212347915,
     30928.954367901584,
     29095.062690792329,
     28426.793310216104,
     26914.31468947696,
     25975.490225306414,
     25435.360392172126,
     24836.868461234892,
     24133.903034204865,
     23626.584694583617,
     23060.447629759739,
     22521.141339972521,
     21986.600362484041,
     21708.632679338189,
     21322.262079304437,
     20787.919953330376,
     20624.359384467432,
     20029.795155720603,
     19853.067069977682,
     19440.537303560875,
     19233.056525067761,
     18875.450036315928,
     18604.189801324806,
     18443.502619621235,
     18096.416501089247,
     17887.377666428139,
     17637.052858147334,
     17348.111435998861,
     17166.150898743901]

Let's plot the reconstruction error to find the kink.

```python
reconstruction_error = np.array(reconstruction_error, dtype = np.float64)
fig = plt.figure(figsize=(10,6))
ax = plt.gca()
ax.plot(k_range, reconstruction_error, '.k-')
ax.grid(True)
ax.set_ylabel('Reconstruction error')
ax.set_xlabel('$k$ - the number of clusters')
ax.set_title('Test Data Reconstruction Error versus $k$')
ax.get_yaxis().set_major_formatter(
    matplotlib.ticker.FuncFormatter(lambda y, pos : format(int(y), ',')))
fig.show()
```

Qualitatively, $100$ looks good to me. Now, let's fit our RBF network, which is a super simple [neural network](https://en.wikipedia.org/wiki/Radial_basis_function_network). We have 1 hidden layer of 100 units with the RBF activation function, and we run a linear regression on that hidden layer.

```python
n_clusters = 100
kMeansModel = cluster.KMeans(n_clusters = n_clusters, init = 'k-means++', algorithm = 'elkan', precompute_distances = True)
kMeansModel.fit(X_scaled)
prototypes = kMeansModel.cluster_centers_
```

The RBF kerneal is

\begin{equation}
\kappa(\mathbf{x}, \mathbf{x}^\prime) = \exp\left(-\frac{\lVert\mathbf{x}-\mathbf{x}^\prime\rVert^2}{2\sigma^2}\right),
\end{equation}

so we'll perform our linear regression with design matrix $\Phi$, where rows of $\Phi$ are

\begin{equation}
\phi(\mathbf{x}) = \begin{pmatrix}
\kappa(\mathbf{x}, \boldsymbol\mu_1) & \kappa(\mathbf{x}, \boldsymbol\mu_2) & \cdots
& \kappa(\mathbf{x}, \boldsymbol\mu_{k})
\end{pmatrix},
\end{equation}

where the $\boldsymbol\mu_j$ for $j = 1,2,\ldots,k$ are our cluster centers. $\sigma^2$ is parameter called *bandwidth* that we'll have to find with cross-validation.

```python
from sklearn import model_selection

class RBFNetwork:
    """
    Implements RBFNetwork as a special case of linear regression.
    """
    def __init__(self, mu, sigma2):
        self.linearRegressor = linear_model.LinearRegression(fit_intercept = True, normalize = False)
        self.mu = mu
        self.sigma2 = sigma2

    def fit(self, X, y):
        Phi = self.rbf_transform(X, self.mu, self.sigma2)
        linearRegressor.fit(Phi, y)
        return self

    def predict(self, X):
        Phi = self.rbf_transform(X, self.mu, self.sigma2)
        return linearRegressor.predict(Phi)

    def score(self, X, y):
        predicted_y = self.predict(X)
        return -(sum((y - predicted_y)**2)/len(y))[0]

    def get_params(self, deep = True):
        return {'mu': self.mu, 'sigma2': self.sigma2}

    def set_params(self, **params):
        if 'mu' in params:
            self.mu = params['mu']
        if 'sigma2' in params:
            self.sigma2 = params['sigma2']
        return self

    @staticmethod
    def rbf(x1, x2, sigma2):
        deltaX = (x1 - x2)
        distX = sum(deltaX*deltaX)
        return np.exp(-distX/2/sigma2)

    @staticmethod
    def phi(x, mu, sigma2):
        return np.apply_along_axis(func1d = RBFNetwork.rbf, axis = 1, arr = mu, x2 = x, sigma2 = sigma2)

    @staticmethod
    def rbf_transform(X, mu, sigma2):
        return np.apply_along_axis(func1d = RBFNetwork.phi, axis = 1, arr = X, mu = mu, sigma2 = sigma2)

bandwidths = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
gridSearchCV = model_selection.GridSearchCV(RBFNetwork(mu = prototypes, sigma2 = 1024),
                                            param_grid = {'sigma2': bandwidths},
                                            cv = model_selection.KFold(n_splits = 5, shuffle = True, random_state = 2016),
                                            n_jobs = 1) ## parallel jobs are broken for a reason that I don't feel like debugging
gridSearchCV.fit(X_scaled, Y_scaled)
cv_results = pd.DataFrame({'bandwidth': bandwidths,
                           'mean_test_score': gridSearchCV.cv_results_['mean_test_score'],
                           'std_test_score': gridSearchCV.cv_results_['std_test_score'],
                           'mean_train_score': gridSearchCV.cv_results_['mean_train_score'],
                           'std_train_score': gridSearchCV.cv_results_['std_train_score']})
cv_results
```

      bandwidth
      mean_test_score
      mean_train_score
      std_test_score
      std_train_score

      0
      1
      -267.152902
      -265.932425
      15.279617
      3.813405

      1
      2
      -188.522188
      -187.648077
      11.168259
      2.790824

      2
      4
      -122.894823
      -122.112400
      6.091327
      1.522200

      3
      8
      -72.431290
      -71.858192
      2.480018
      0.620141

      4
      16
      -40.163809
      -39.825444
      0.767083
      0.193418

      5
      32
      -25.264711
      -25.057333
      0.227940
      0.056388

      6
      64
      -20.167603
      -20.004292
      0.097739
      0.025807

      7
      128
      -18.716779
      -18.568064
      0.192818
      0.049968

      8
      256
      -18.422612
      -18.277355
      0.228230
      0.058791

      9
      512
      -18.432984
      -18.287904
      0.238813
      0.061415

```python
fig = plt.figure(figsize=(10, 6))
ax = fig.gca()
test_error_bars = ax.errorbar(cv_results['bandwidth'], cv_results['mean_test_score'], cv_results['std_test_score'],
                              linestyle='-', marker='.', color='b', capsize=5)
train_error_bars = ax.errorbar(cv_results['bandwidth'], cv_results['mean_train_score'], cv_results['std_train_score'],
                              linestyle='-', marker='.', color='r', capsize=5)
ax.grid(True)
ax.legend([test_error_bars, train_error_bars], ['Test Set', 'Training Set'], loc='lower right')
ax.set_xscale('log', basex = 2)
ax.set_xlabel('Bandwidth ($\sigma^2$)')
ax.set_ylabel('Score (negative MSE)')
ax.set_title('Score versus Bandwidth')
fig.show()
```

It looks like the best model has a bandwidth of about $2^{8}$. Let's compute its SMSE.

```python
rBFNetwork = RBFNetwork(mu = prototypes, sigma2 = 2**8)
rBFNetwork.fit(X_scaled, Y_scaled)
-rBFNetwork.score(X_scaled_test, Y_scaled_test)/np.var(Y_scaled, ddof=0)
```

    0.042844306703931787

Seems that we can only get down to an SMSE of 0.043. The next step is to try a neural network. This has more parameters to choose. In particular, we're interested in the number of hidden units, layers, and the L2 regularization parameter.

## Neural Networks with TensorFlow

Since I work at Google, I thought that I would demonstrate constructing a neural network in [TensorFlow](https://www.tensorflow.org/). Since I'm doing this with the intent of learning the details of the models, I'm going to eschew the high-level APIs and construct the graph mostly by hand. Much of the code is imported from [`sarco_train.py`](https://github.com/ppham27/MLaPP-solutions/blob/master/chap16/sarco_train.py).

First, we'll set up our features which need to normalized. We make it so that the input has mean 0 and variance 1.

```python
import tensorflow as tf
from sarco_train import create_feature_columns

feature_columns = create_feature_columns(pd.concat([training_data, test_data]))

def get_feature_data(data, feature_columns):
    graph = tf.Graph()
    with graph.as_default():
        features = {column: tf.constant(column_data, dtype=tf.float32)
                    for column, column_data in data.to_dict('list').items()}
        input_layer = tf.feature_column.input_layer(
            features=features, feature_columns=feature_columns)
    with tf.Session(graph=graph) as sess:
        input_data = sess.run(input_layer)
    return input_data, data[TARGET_COLUMN].astype(np.float32)

training_input_data, training_target_data = get_feature_data(training_data, feature_columns)
test_input_data, test_target_data = get_feature_data(test_data, feature_columns)
```

## Building the Graph

Everything in TensorFlow is a graph. We build build it layer by layer.

### Input Layer

Here, we extract the features from the training data and test data and apply the feature columns to the two datasets.

### Inference Layer

Now, we can think about the inference layer. To keep things rather simple, I'll only use two layers. The weights and biases are initialize with noise, ReLU is used as the activation function. The first layer has 70 units, and the second layer has 50 units. Since we have an input of 21 features, the total number of weight parameters in a fully connected network are $21 \times 70 + 70 \times 50 + 50 \times 1$, and we have also have $70 + 50 + 1$ bias parameters. The number of units was chosen arbitrarily. I was too lazy to implement and cross validation strategy.

### Loss Layer

For loss, we minimize mean squared error. This is we'll compare models.

```python
from sarco_train import inference, HIDDEN1_UNITS, HIDDEN2_UNITS

eval_graph = tf.Graph()
with eval_graph.as_default():
    input = tf.placeholder(dtype=tf.float32, shape=(None, len(feature_columns)))
    prediction = inference(input, HIDDEN1_UNITS, HIDDEN2_UNITS)

    labels = tf.placeholder(dtype=tf.float32, shape=(None,))
    mse = tf.losses.mean_squared_error(labels=labels, predictions=prediction, reduction=tf.losses.Reduction.MEAN)

    tf.summary.scalar('mse', mse)

    summary_op = tf.summary.merge_all()
# Reset evaluation process
last_checkpoint_summarized = None
```

## Training the Model

Training the model is done in [`sarco_train.py`](https://github.com/ppham27/MLaPP-solutions/blob/master/chap16/sarco_train.py). The script was run with the command:

`python3 sarco_train.py --restart --log_level=INFO --log_dir /Users/phillypham/tmp/sarco/logs`

Input is fetched from a queue to avoid blocking the thread doing the training. 1024 examples are processed in each training step. In addition to minimize mean squared error, we add $L^1$ and $L^2$ regularization to deal with the large number of parameters. The scale parameter for $L^1$ was $1.0$. The scale parameter for $L^2$ is $14.0$. The variation of gradient descent that we'll use is [Adam Optimizer](https://arxiv.org/abs/1412.6980) with a learning rate of $0.01$.

While the training script is running, this code runs in a separate process to periodically evaluate the model. We calculate mean squared error against the entire training set and entire test set. We can see these plotted by running `tensorboard --logdir=/Users/phillypham/tmp/sarco/logs` and visiting [TensorBoard](http://localhost:6006).

Since we're using a [Monitored Session](https://www.tensorflow.org/versions/master/api_docs/python/tf/train/MonitoredSession), we can continue training from where left off by running

`python3 sarco_train.py --log_level=INFO --log_dir /Users/phillypham/tmp/sarco/logs`

and restarting the evaluation code below.

```python
from sarco_train import CHECKPOINT_INTERVAL_SECS

CHECKPOINT_DIR = '/Users/phillypham/tmp/sarco/logs'
EVAL_DIR = CHECKPOINT_DIR + '/eval'

with eval_graph.as_default():
    saver = tf.train.Saver(tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES))

    training_writer = tf.summary.FileWriter(EVAL_DIR + '/training', eval_graph)
    test_writer = tf.summary.FileWriter(EVAL_DIR + '/test', eval_graph)
    with tf.Session() as sess:
        def write_summary(step):
            def write_mse(writer, features, target):
                summary = sess.run(summary_op, feed_dict={input: features, labels: target})
                writer.add_summary(summary, step)
                writer.flush()
            tf.logging.info('Writing summary for step {:d}'.format(step))
            write_mse(test_writer, test_input_data, test_target_data)
            write_mse(training_writer, training_input_data, training_target_data)
        while True:
            checkpoint_state = tf.train.get_checkpoint_state(CHECKPOINT_DIR)
            checkpoint_path = checkpoint_state.model_checkpoint_path
            # Stop running when no more new checkpoints are being generated.
            if checkpoint_path == last_checkpoint_summarized:
                break
            saver.restore(sess, checkpoint_state.model_checkpoint_path)
            global_step = int(checkpoint_state.model_checkpoint_path.split('/')[-1].split('-')[-1])
            write_summary(global_step)
            last_checkpoint_summarized = checkpoint_path
            time.sleep(CHECKPOINT_INTERVAL_SECS)
        tf.logging.info('Training stopped.')
```

    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-1
    INFO:tensorflow:Writing summary for step 1
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-1678
    INFO:tensorflow:Writing summary for step 1678
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-3364
    INFO:tensorflow:Writing summary for step 3364
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-5052
    INFO:tensorflow:Writing summary for step 5052
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-6742
    INFO:tensorflow:Writing summary for step 6742
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-8430
    INFO:tensorflow:Writing summary for step 8430
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-10121
    INFO:tensorflow:Writing summary for step 10121
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-11806
    INFO:tensorflow:Writing summary for step 11806
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-13495
    INFO:tensorflow:Writing summary for step 13495
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-15180
    INFO:tensorflow:Writing summary for step 15180
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-16860
    INFO:tensorflow:Writing summary for step 16860
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-18540
    INFO:tensorflow:Writing summary for step 18540
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-20217
    INFO:tensorflow:Writing summary for step 20217
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-21918
    INFO:tensorflow:Writing summary for step 21918
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-23602
    INFO:tensorflow:Writing summary for step 23602
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-25280
    INFO:tensorflow:Writing summary for step 25280
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-26960
    INFO:tensorflow:Writing summary for step 26960
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-28642
    INFO:tensorflow:Writing summary for step 28642
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-30323
    INFO:tensorflow:Writing summary for step 30323
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-32005
    INFO:tensorflow:Writing summary for step 32005
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-33683
    INFO:tensorflow:Writing summary for step 33683
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-35363
    INFO:tensorflow:Writing summary for step 35363
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-37045
    INFO:tensorflow:Writing summary for step 37045
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-38732
    INFO:tensorflow:Writing summary for step 38732
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-40423
    INFO:tensorflow:Writing summary for step 40423
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-42118
    INFO:tensorflow:Writing summary for step 42118
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-43804
    INFO:tensorflow:Writing summary for step 43804
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-45490
    INFO:tensorflow:Writing summary for step 45490
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-47172
    INFO:tensorflow:Writing summary for step 47172
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-48858
    INFO:tensorflow:Writing summary for step 48858
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-50543
    INFO:tensorflow:Writing summary for step 50543
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-52230
    INFO:tensorflow:Writing summary for step 52230
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-53916
    INFO:tensorflow:Writing summary for step 53916
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-55598
    INFO:tensorflow:Writing summary for step 55598
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-57287
    INFO:tensorflow:Writing summary for step 57287
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-58973
    INFO:tensorflow:Writing summary for step 58973
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-60658
    INFO:tensorflow:Writing summary for step 60658
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-62341
    INFO:tensorflow:Writing summary for step 62341
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-64022
    INFO:tensorflow:Writing summary for step 64022
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-65707
    INFO:tensorflow:Writing summary for step 65707
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-67391
    INFO:tensorflow:Writing summary for step 67391
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-69078
    INFO:tensorflow:Writing summary for step 69078
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-70766
    INFO:tensorflow:Writing summary for step 70766
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-72450
    INFO:tensorflow:Writing summary for step 72450
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-74140
    INFO:tensorflow:Writing summary for step 74140
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-75830
    INFO:tensorflow:Writing summary for step 75830
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-77519
    INFO:tensorflow:Writing summary for step 77519
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-79229
    INFO:tensorflow:Writing summary for step 79229
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-80912
    INFO:tensorflow:Writing summary for step 80912
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-82598
    INFO:tensorflow:Writing summary for step 82598
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-84285
    INFO:tensorflow:Writing summary for step 84285
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-85968
    INFO:tensorflow:Writing summary for step 85968
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-87651
    INFO:tensorflow:Writing summary for step 87651
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-89335
    INFO:tensorflow:Writing summary for step 89335
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-91024
    INFO:tensorflow:Writing summary for step 91024
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-92710
    INFO:tensorflow:Writing summary for step 92710
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-94397
    INFO:tensorflow:Writing summary for step 94397
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-96088
    INFO:tensorflow:Writing summary for step 96088
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-97777
    INFO:tensorflow:Writing summary for step 97777
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-99478
    INFO:tensorflow:Writing summary for step 99478
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-101169
    INFO:tensorflow:Writing summary for step 101169
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-102850
    INFO:tensorflow:Writing summary for step 102850
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-104538
    INFO:tensorflow:Writing summary for step 104538
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-106225
    INFO:tensorflow:Writing summary for step 106225
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-107909
    INFO:tensorflow:Writing summary for step 107909
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-109595
    INFO:tensorflow:Writing summary for step 109595
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-111286
    INFO:tensorflow:Writing summary for step 111286
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-112972
    INFO:tensorflow:Writing summary for step 112972
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-114656
    INFO:tensorflow:Writing summary for step 114656
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-116338
    INFO:tensorflow:Writing summary for step 116338
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-118021
    INFO:tensorflow:Writing summary for step 118021
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-119704
    INFO:tensorflow:Writing summary for step 119704
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-121389
    INFO:tensorflow:Writing summary for step 121389
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-123079
    INFO:tensorflow:Writing summary for step 123079
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-124769
    INFO:tensorflow:Writing summary for step 124769
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-126468
    INFO:tensorflow:Writing summary for step 126468
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-128166
    INFO:tensorflow:Writing summary for step 128166
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-129857
    INFO:tensorflow:Writing summary for step 129857
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-131561
    INFO:tensorflow:Writing summary for step 131561
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-133266
    INFO:tensorflow:Writing summary for step 133266
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-134949
    INFO:tensorflow:Writing summary for step 134949
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-136636
    INFO:tensorflow:Writing summary for step 136636
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-138333
    INFO:tensorflow:Writing summary for step 138333
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-140023
    INFO:tensorflow:Writing summary for step 140023
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-141702
    INFO:tensorflow:Writing summary for step 141702
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-143392
    INFO:tensorflow:Writing summary for step 143392
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-145086
    INFO:tensorflow:Writing summary for step 145086
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-146778
    INFO:tensorflow:Writing summary for step 146778
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-148474
    INFO:tensorflow:Writing summary for step 148474
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-150174
    INFO:tensorflow:Writing summary for step 150174
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-151871
    INFO:tensorflow:Writing summary for step 151871
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-153556
    INFO:tensorflow:Writing summary for step 153556
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-155239
    INFO:tensorflow:Writing summary for step 155239
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-156921
    INFO:tensorflow:Writing summary for step 156921
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-158602
    INFO:tensorflow:Writing summary for step 158602
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-160279
    INFO:tensorflow:Writing summary for step 160279
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-161977
    INFO:tensorflow:Writing summary for step 161977
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-163659
    INFO:tensorflow:Writing summary for step 163659
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-165343
    INFO:tensorflow:Writing summary for step 165343
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-167032
    INFO:tensorflow:Writing summary for step 167032
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-168715
    INFO:tensorflow:Writing summary for step 168715
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-170392
    INFO:tensorflow:Writing summary for step 170392
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-172072
    INFO:tensorflow:Writing summary for step 172072
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-173748
    INFO:tensorflow:Writing summary for step 173748
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-175438
    INFO:tensorflow:Writing summary for step 175438
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-177123
    INFO:tensorflow:Writing summary for step 177123
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-178806
    INFO:tensorflow:Writing summary for step 178806
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-180493
    INFO:tensorflow:Writing summary for step 180493
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-182177
    INFO:tensorflow:Writing summary for step 182177
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-183862
    INFO:tensorflow:Writing summary for step 183862
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-185550
    INFO:tensorflow:Writing summary for step 185550
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-187241
    INFO:tensorflow:Writing summary for step 187241
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-188939
    INFO:tensorflow:Writing summary for step 188939
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-190640
    INFO:tensorflow:Writing summary for step 190640
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-192328
    INFO:tensorflow:Writing summary for step 192328
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-194023
    INFO:tensorflow:Writing summary for step 194023
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-195718
    INFO:tensorflow:Writing summary for step 195718
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-197414
    INFO:tensorflow:Writing summary for step 197414
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-199109
    INFO:tensorflow:Writing summary for step 199109
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-200812
    INFO:tensorflow:Writing summary for step 200812
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-202527
    INFO:tensorflow:Writing summary for step 202527
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-204216
    INFO:tensorflow:Writing summary for step 204216
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-205901
    INFO:tensorflow:Writing summary for step 205901
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-207596
    INFO:tensorflow:Writing summary for step 207596
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-209288
    INFO:tensorflow:Writing summary for step 209288
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-210977
    INFO:tensorflow:Writing summary for step 210977
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-212672
    INFO:tensorflow:Writing summary for step 212672
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-214366
    INFO:tensorflow:Writing summary for step 214366
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-216059
    INFO:tensorflow:Writing summary for step 216059
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-217750
    INFO:tensorflow:Writing summary for step 217750
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-219444
    INFO:tensorflow:Writing summary for step 219444
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-221136
    INFO:tensorflow:Writing summary for step 221136
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-222833
    INFO:tensorflow:Writing summary for step 222833
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-224524
    INFO:tensorflow:Writing summary for step 224524
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-226225
    INFO:tensorflow:Writing summary for step 226225
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-227929
    INFO:tensorflow:Writing summary for step 227929
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-229629
    INFO:tensorflow:Writing summary for step 229629
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-231323
    INFO:tensorflow:Writing summary for step 231323
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-233027
    INFO:tensorflow:Writing summary for step 233027
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-234729
    INFO:tensorflow:Writing summary for step 234729
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-236423
    INFO:tensorflow:Writing summary for step 236423
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-238118
    INFO:tensorflow:Writing summary for step 238118
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-239812
    INFO:tensorflow:Writing summary for step 239812
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-241514
    INFO:tensorflow:Writing summary for step 241514
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-243209
    INFO:tensorflow:Writing summary for step 243209
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-244904
    INFO:tensorflow:Writing summary for step 244904
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-246592
    INFO:tensorflow:Writing summary for step 246592
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-248282
    INFO:tensorflow:Writing summary for step 248282
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-249971
    INFO:tensorflow:Writing summary for step 249971
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-251664
    INFO:tensorflow:Writing summary for step 251664
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-253358
    INFO:tensorflow:Writing summary for step 253358
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-255057
    INFO:tensorflow:Writing summary for step 255057
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-256742
    INFO:tensorflow:Writing summary for step 256742
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-258426
    INFO:tensorflow:Writing summary for step 258426
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-260111
    INFO:tensorflow:Writing summary for step 260111
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-261796
    INFO:tensorflow:Writing summary for step 261796
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-263477
    INFO:tensorflow:Writing summary for step 263477
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-265166
    INFO:tensorflow:Writing summary for step 265166
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-266868
    INFO:tensorflow:Writing summary for step 266868
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-268547
    INFO:tensorflow:Writing summary for step 268547
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-270228
    INFO:tensorflow:Writing summary for step 270228
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-271914
    INFO:tensorflow:Writing summary for step 271914
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-273596
    INFO:tensorflow:Writing summary for step 273596
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-275275
    INFO:tensorflow:Writing summary for step 275275
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-276956
    INFO:tensorflow:Writing summary for step 276956
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-278637
    INFO:tensorflow:Writing summary for step 278637
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-280318
    INFO:tensorflow:Writing summary for step 280318
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-282002
    INFO:tensorflow:Writing summary for step 282002
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-283699
    INFO:tensorflow:Writing summary for step 283699
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-285401
    INFO:tensorflow:Writing summary for step 285401
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-287103
    INFO:tensorflow:Writing summary for step 287103
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-288801
    INFO:tensorflow:Writing summary for step 288801
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-290504
    INFO:tensorflow:Writing summary for step 290504
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-292212
    INFO:tensorflow:Writing summary for step 292212
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-293921
    INFO:tensorflow:Writing summary for step 293921
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-295627
    INFO:tensorflow:Writing summary for step 295627
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-297332
    INFO:tensorflow:Writing summary for step 297332
    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-298176
    INFO:tensorflow:Writing summary for step 298176
    INFO:tensorflow:Training stopped.

## Monitoring the training process

In TensorBoard, I've been summarizing loss and the mean squared error on checkpoints as the model is training. Here's the loss plotted.

You can see that the model has more or less by converged 50,000 training steps. Because we're sampling batches of data, there's a lot of noise.

The loss includes the regularization penalty, so when evaluating the model on the test data, it may be more helpful to look at mean squared error. The various mean squared errors are:
- **Orange**: Batch that we're training on
- **Blue**: The test set
- **Purple**: The training set

## Selecting a model

Of course, this isn't very helpful because of how noisy the MSE of the training batch set is. It's more helpful to look at just the MSEs from the evaluation process.

After the model converged, there's a lot of random noise. We may possibly be overfitting by training this long. The MSE on the training and test set stay pretty close together, however. It looks like the lowest MSE is obtained by the model at step 188,939. Let's calculate the SMSE for this model.

```python
SELECTED_MODEL = CHECKPOINT_DIR + '/model.ckpt-188939'

with eval_graph.as_default():
    with tf.Session() as sess:
        saver.restore(sess, SELECTED_MODEL)
        test_mse = sess.run(mse, feed_dict={input: test_input_data, labels: test_target_data})
        log_mse(test_mse)
```

    INFO:tensorflow:Restoring parameters from /Users/phillypham/tmp/sarco/logs/model.ckpt-188939
    MSE: 9.1203241348
    SMSE: 0.0217773683

Nice. Neural networks achieve the best standarded mean squared error. Of course, training this model took the longest, and the details of why it works are already obscure after only two layers.


Table of Contents

1  Intro to Pytorch

1.1  Pytorch tensors

1.2  Pytorch Autograd

1.3  torch.nn module

2  Linear Regression with Pytorch

2.1  Gradient Descent with Pytorch

2.2  Simplified GD Loop

2.2.1  Models in Pytorch

3  Logistic Regression

4  Dataset and Data loaders

5  Two layer neural network

5.1  Lab

6  References

```python
# import pytorch libraries
%matplotlib inline
import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
```

# Intro to Pytorch

PyTorch consists of 4 main packages:
* torch: a general purpose array library similar to Numpy that can do computations on GPU
* torch.autograd: a package for automatically obtaining gradients - automatic diffrentiation of gradients
* torch.nn: a neural net library with common layers and cost functions
* torch.optim: an optimization package with common optimization algorithms like SGD, Adam, etc

## Pytorch tensors
Like Numpy tensors but can utilize GPUs to accelerate its numerical computations.

```python
# Create random tensor
N = 5
x = torch.randn(N, 10).type(torch.FloatTensor)
```

torch.randn - Returns a tensor filled with random numbers from a normal distribution with mean 0 and variance 1 (also called the standard normal distribution).

```python
ttom = torch.tensor([[1,1,1,1],[2,2,2,2],[3,3,3,3]], dtype=torch.float32)
ttom

```

    tensor([[1., 1., 1., 1.],
            [2., 2., 2., 2.],
            [3., 3., 3., 3.]])

```python
ttom.size()

```

    torch.Size([3, 4])

```python
len(ttom.shape)

```

    2

```python
torch.set_printoptions(threshold=5000)
x
```

    tensor([[ 1.1340,  0.7798,  0.5917,  0.7891, -0.4278, -1.0299,  0.2895,  0.3383,
              0.9109,  1.5371],
            [-2.0633, -0.0342, -0.1908, -1.5765, -0.1915, -0.2203, -0.8096,  0.6704,
              0.0208,  0.7704],
            [-0.5604, -0.2043,  0.3470, -0.9613,  0.7098, -0.8015,  0.5235, -0.3343,
             -0.5754, -2.4173],
            [ 1.1895,  2.2036, -0.4244,  0.6212, -0.3357, -0.1387, -0.3092,  0.0474,
              0.4143, -1.0411],
            [ 2.5717,  0.2043,  0.0410, -0.1242,  0.3913, -1.0310, -0.3673, -1.1306,
             -0.2102,  0.9751]])

```python
# reshaping of tensors using .view()
x.view(1,-1) #-1 makes torch infer the second dim
```

    tensor([[ 1.1340,  0.7798,  0.5917,  0.7891, -0.4278, -1.0299,  0.2895,  0.3383,
              0.9109,  1.5371, -2.0633, -0.0342, -0.1908, -1.5765, -0.1915, -0.2203,
             -0.8096,  0.6704,  0.0208,  0.7704, -0.5604, -0.2043,  0.3470, -0.9613,
              0.7098, -0.8015,  0.5235, -0.3343, -0.5754, -2.4173,  1.1895,  2.2036,
             -0.4244,  0.6212, -0.3357, -0.1387, -0.3092,  0.0474,  0.4143, -1.0411,
              2.5717,  0.2043,  0.0410, -0.1242,  0.3913, -1.0310, -0.3673, -1.1306,
             -0.2102,  0.9751]])

## Pytorch Autograd
The autograd package in PyTorch provides classes and functions implementing automatic differentiation of arbitrary scalar valued function. For example, the gradient of the error with respect to all parameters.

In order for this to happen we need to declare our paramerers as Tensors with the requires_grad=True keyword. Here is an example:

```python
x = torch.tensor([1., 2., 3., 4., 5., 6., 7., 8., 9., 10.], requires_grad=True) # These are parameters
```

```python
x.grad
```

```python
L = (2*x**2+1).sum()
L
```

    tensor(780., grad_fn=<SumBackward0>)

```python
L.backward() # computes the grad of L with respect to x
```

```latex
%%latex
\begin{align}
\newline
\dfrac{dy}{dx}(2x+1) = 2
\newline
\newline
\dfrac{dy}{dx}({2x^2+1}) = 4x
\newline
\newline
\end{align}
```

\begin{align}
\newline
\dfrac{dy}{dx}(2x+1) = 2
\newline
\newline
\dfrac{dy}{dx}({2x^2+1}) = 4x
\newline
\newline
\end{align}

```python
x.grad
```

    tensor([ 4.,  8., 12., 16., 20., 24., 28., 32., 36., 40.])

## torch.nn module
A neural net library with common layers and cost functions

`nn.Linear(5, 3)` creates a linear transformation ($A\cdot X+b$) of a $N \times 5$ matrix into a $N \times 3$ matrix, where N can be anything (number of observations).

```class torch.nn.Linear(in_features, out_features, bias=True)```

Applies a linear transformation to the incoming data: y=x$A^T$+b

https://pytorch.org/docs/stable/nn.html#linear

```python
D = 5 # number of input featutes
M = 3 # neurons in the first hidden layer
linear_map = nn.Linear(D, M)
```

```python
# parameters are initialized randomly
[p for p in linear_map.parameters()]
```

    [Parameter containing:
     tensor([[-0.3488, -0.0565, -0.3512,  0.1248, -0.2431],
             [-0.0775,  0.1706, -0.3613, -0.0244, -0.2534],
             [ 0.0831,  0.1206,  0.2780,  0.3938, -0.2368]], requires_grad=True),
     Parameter containing:
     tensor([-0.2345, -0.2503, -0.4466], requires_grad=True)]

```python
[p.shape for p in linear_map.parameters()]
```

    [torch.Size([3, 5]), torch.Size([3])]

#  Linear Regression with Pytorch

The goal of linear regression is to fit a line to a set of points.

```python
# Here we generate some fake data
def lin(a,b,x): return a*x+b

def gen_fake_data(n, a, b):
    x = np.random.uniform(0,1,n)
    y = lin(a,b,x) + 0.1 * np.random.normal(0,3,n)
    return x, y

x, y = gen_fake_data(500, 3., 8.) # a = 3 b = 8
```

```python
import matplotlib.pyplot as plt
plt.scatter(x,y, s=8); plt.xlabel("x"); plt.ylabel("y");
```

You want to find **parameters** (weights) $a$ and $b$ such that you minimize the *error* between the points and the line $a\cdot x + b$. Note that here $a$ and $b$ are unknown. For a regression problem the most common *error function* or *loss function* is the **mean squared error** ($\sum_i (\hat{y}_i - y_i)^2$).

```python
def mse(y_hat, y): return ((y_hat - y) ** 2).mean()
```

Suppose we believe $a = 10$ and $b = 5$ then we can compute `y_hat` which is our *prediction* and then compute our error.

```python
y_hat = lin(10,5,x)
mse(y_hat, y)
```

    4.129093168542686

```python
def mse_loss(a, b, x, y): return mse(lin(a,b,x), y)
```

```python
mse_loss(10, 5, x, y)
```

    4.129093168542686

So far we have specified the *model* (linear regression) and the *evaluation criteria* (or *loss function*). Now we need to handle *optimization*; that is, how do we find the best values for $a$ and $b$? How do we find the best *fitting* linear regression.

## Gradient Descent with Pytorch

For a fixed dataset $x$ and $y$ `mse_loss(a,b)` is a function of $a$ and $b$. We would like to find the values of $a$ and $b$ that minimize that function.

**Gradient descent** is an algorithm that minimizes functions. Given a function defined by a set of parameters, gradient descent starts with an initial set of parameter values and iteratively moves toward a set of parameter values that minimize the function. This iterative minimization is achieved by taking steps in the negative direction of the function gradient.

Here is gradient descent implemented in [PyTorch](http://pytorch.org/).

```python
# generate some more data
x, y = gen_fake_data(10000, 3., 8.)
x.shape, y.shape
```

    ((10000,), (10000,))

```python
# Wrap x and y as tensor
x = torch.tensor(x)
y = torch.tensor(y)
```

```python
# Create random Tensors for weights, and wrap them in tensors.
# Setting requires_grad=True indicates that we want to compute gradients with
# respect to these tensors during the backward pass.
a, b = np.random.randn(1), np.random.randn(1)
a = torch.tensor(a, requires_grad=True)
b = torch.tensor(b, requires_grad=True)
a,b
```

    (tensor([-0.4721], dtype=torch.float64, requires_grad=True),
     tensor([0.6668], dtype=torch.float64, requires_grad=True))

```python
learning_rate = 1e-3
for t in range(10000):
    # Forward pass: compute predicted y using operations on Variables
    loss = mse_loss(a,b,x,y)
    if t % 1000 == 0: print(loss.item())

    # Computes the gradient of loss with respect to all Variables with requires_grad=True.
    # After this call a.grad and b.grad will be Variables holding the gradient
    # of the loss with respect to a and b respectively
    loss.backward()

    # Update a and b using gradient descent; a.data and b.data are Tensors,
    # a.grad and b.grad are Variables and a.grad.data and b.grad.data are Tensors
    a.data -= learning_rate * a.grad.data
    b.data -= learning_rate * b.grad.data

    # Zero the gradients
    a.grad.data.zero_()
    b.grad.data.zero_()
```

    83.45272457689768
    0.6133767913493735
    0.09863587239090943
    0.09393122743427093
    0.0927350799610742
    0.09183074907429083
    0.09113581144951306
    0.09060171246385135
    0.09019122665859797
    0.08987574469311128

```python
print(a,b)
```

    tensor([3.0964], dtype=torch.float64, requires_grad=True) tensor([7.9475], dtype=torch.float64, requires_grad=True)

## Simplified GD Loop

```python
# linear tranformation with input dimension=1 and output dimension=1
nn.Linear(1, 1)
```

    Linear(in_features=1, out_features=1, bias=True)

### Models in Pytorch

```python
# simple way of specifying a linear regression model
model = torch.nn.Sequential(
    nn.Linear(1, 1),
)
model
```

    Sequential(
      (0): Linear(in_features=1, out_features=1, bias=True)
    )

```python
# equivalent way of specifiying the same model
class LinearRegression(nn.Module):
    def __init__(self):
        super(LinearRegression, self).__init__()
        self.lin = nn.Linear(1, 1)

    def forward(self, x):
        x = self.lin(x)
        return x
model =  LinearRegression()
```

```python
# note here we have just two parameters, why?
print([p for p in model.parameters()])
```

    [Parameter containing:
    tensor([[0.7945]], requires_grad=True), Parameter containing:
    tensor([-0.3674], requires_grad=True)]

```python
x, y = gen_fake_data(10000, 3., 8.)
x = torch.tensor(x).float()
y = torch.tensor(y).float()
x.shape
```

    torch.Size([10000])

```python
# you have to be careful with the dimensions that your model is expecting
x1 = torch.unsqueeze(x, 1)
x1.shape
```

    torch.Size([10000, 1])

```python
y_hat = model(x1)
print(y_hat)
```

    tensor([[-0.3229],
            [ 0.3150],
            [-0.0714],
            ...,
            [ 0.2098],
            [-0.2030],
            [-0.0680]], grad_fn=<AddmmBackward>)

```python
# Use the optim package to define an Optimizer that will update the weights of
# the model for us. Here we will use Adam
learning_rate = 0.1
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
```

```python
for t in range(10000):
    # Forward pass: compute predicted y using operations on Variables
    y_hat = model(x1)
    loss = F.mse_loss(y_hat, y.unsqueeze(1))
    if t % 1000 == 0: print(loss.item())

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables
    optimizer.zero_grad()
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its
    # parameters
    optimizer.step()
```

    90.24227905273438
    0.08916361629962921
    0.08915530145168304
    0.08915530145168304
    0.08915530145168304
    0.08915530145168304
    0.08915530145168304
    0.08915530145168304
    0.08915530145168304
    0.08915530145168304

```python
print([p for p in model.parameters()])
```

    [Parameter containing:
    tensor([[2.9857]], requires_grad=True), Parameter containing:
    tensor([8.0094], requires_grad=True)]

# Logistic Regression

```python
# generating fake data
# Here we generate some fake data
def lin(a,b,x): return a*x+b

def gen_logistic_fake_data(n, a, b):
    x = np.random.uniform(-20,20, (n, 2))
    x2_hat = lin(a,b, x[:,0])
    y = x[:,1] > x2_hat
    return x, y.astype(int)

x, y = gen_logistic_fake_data(100, 1., 0.5)
```

```python
y
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0,
           1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1,
           1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1,
           1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1,
           1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1])

```python
t = np.arange(-20, 20, 0.2)
import matplotlib.pyplot as plt
plt.scatter(x[:,0],x[:,1],c=y, s=8);
plt.xlabel("x1"); plt.ylabel("x2");
plt.plot(t, t + 0.5, 'r--')
```

```python
x = torch.tensor(x).float()
y = torch.tensor(y).float()
```

```python
model = torch.nn.Sequential(
    torch.nn.Linear(2, 1),
)
model
```

    Sequential(
      (0): Linear(in_features=2, out_features=1, bias=True)
    )

```python
model(x).shape
```

    torch.Size([100, 1])

```python
x, y = gen_logistic_fake_data(10000, 1., 0.5)
x = torch.tensor(x).float()
y = torch.tensor(y).float()
```

```python
learning_rate = 0.1
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
```

```python
for t in range(10000):
    # Forward pass: compute predicted y using operations on Variables
    y_hat = model(x)
    loss = F.binary_cross_entropy(F.sigmoid(y_hat), y.unsqueeze(1))
    if t % 1000 == 0: print(loss.item())

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables
    optimizer.zero_grad()
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its
    # parameters
    optimizer.step()
```

    /Users/chaitanya/Desktop/C/PyCharmProjects/Pytorch-DeepLearn/deep-learning-data-institute/lib/python3.7/site-packages/torch/nn/functional.py:1332: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
      warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")

    0.2937122583389282
    0.010437630116939545
    0.006919992156326771
    0.005215308163315058
    0.0041087307035923
    0.0032962923869490623
    0.002663073828443885
    0.0021553158294409513
    0.0017438260838389397
    0.0014104096917435527

```python
print([p for p in model.parameters()])
```

    [Parameter containing:
    tensor([[-28.6590,  28.6310]], requires_grad=True), Parameter containing:
    tensor([-14.4690], requires_grad=True)]

# Dataset and Data loaders

Nearly all of deep learning is powered by one very important algorithm: **stochastic gradient descent (SGD)**. SGD can be seeing as an approximation of **gradient descent** (GD). In GD you have to run through *all* the samples in your training set to do a single itaration. In SGD you use *only one* or *a subset*  of training samples to do the update for a parameter in a particular iteration. The subset use in every iteration is called a **batch** or **minibatch**.

```python
model2 = torch.nn.Sequential(
    torch.nn.Linear(1, 1),
)
```

```python
from torch.utils.data import Dataset, DataLoader
```

```python
def lin(a,b,x): return a*x+b

def gen_fake_data(n, a, b):
    x = np.random.uniform(0,1,n)
    y = lin(a,b,x) + 0.1 * np.random.normal(0,3,n)
    return x.astype(np.float32), y.astype(np.float32)

# create a dataset
class RegressionDataset(Dataset):
    def __init__(self, a=3, b=8, n=10000):
        x, y = gen_fake_data(n, a, b)
        x = torch.from_numpy(x).unsqueeze(1)
        y = torch.from_numpy(y)
        self.x, self.y = x, y

    def __len__(self):
        return len(self.y)

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

fake_dataset = RegressionDataset()
```

Next we are going to create a data loader. The data loader provides the following features:
* Batching the data
* Shuffling the data
* Load the data in parallel using multiprocessing workers.

```python
dataloader = DataLoader(fake_dataset, batch_size=1000, shuffle=True)
x, y = next(iter(dataloader))
```

```python
x.shape, y.shape
```

    (torch.Size([1000, 1]), torch.Size([1000]))

```python
learning_rate = 0.1
optimizer = torch.optim.Adam(model2.parameters(), lr=learning_rate)
```

```python
for t in range(200):
    for i, (x, y) in enumerate(dataloader):

        y_hat = model2(x)
        loss = F.mse_loss(y_hat, y.unsqueeze(1))

        optimizer.zero_grad()
        loss.backward()

        optimizer.step()
    if t % 50 == 0: print(loss.item())
```

    68.08111572265625
    0.09767811745405197
    0.09069585055112839
    0.08714727312326431

```python
print([p for p in model2.parameters()])
```

    [Parameter containing:
    tensor([[2.9965]], requires_grad=True), Parameter containing:
    tensor([8.0014], requires_grad=True)]

# Two layer neural network

```python
# generating fake data
# Here we generate some fake data
def sigmoid(x):
    return 1/(1 + np.exp(-x))

def gen_nn_fake_data(n):
    x = np.random.uniform(0,10, (n, 2))
    x1 = x[:,0]
    x2 = x[:,1]
    score1 = sigmoid(-x1 - 8* x2 + 50)
    score2 = sigmoid(-7*x1 - 2* x2 + 50)
    score3 = 2* score1 + 3*score2 - 0.1
    y = score3 < 0
    return x, y.astype(int)

x, y = gen_nn_fake_data(500)
```

```python
import matplotlib.pyplot as plt
plt.scatter(x[:,0],x[:,1],c=y, s=8);
plt.xlabel("x1"); plt.ylabel("x2");
```

```python
model = torch.nn.Sequential(
    torch.nn.Linear(2, 2),
    torch.nn.Sigmoid(),
    torch.nn.Linear(2, 1)
)
model
```

    Sequential(
      (0): Linear(in_features=2, out_features=2, bias=True)
      (1): Sigmoid()
      (2): Linear(in_features=2, out_features=1, bias=True)
    )

```python
x, y = gen_nn_fake_data(10000)
x = torch.tensor(x).float()
y = torch.tensor(y).float()
```

```python
learning_rate = 0.01
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
```

```python
for t in range(10000):
    # Forward pass: compute predicted y using operations on Variables
    y_hat = model(x)
    loss = F.binary_cross_entropy(F.sigmoid(y_hat), y.unsqueeze(1))
    if t % 1000 == 0: print(loss.item())

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables
    optimizer.zero_grad()
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its
    # parameters
    optimizer.step()
```

    0.7679895758628845
    0.5014307498931885
    0.5011200308799744
    0.5010212659835815
    0.5009831786155701
    0.5009657740592957
    0.5009567141532898
    0.5009514689445496
    0.5009486079216003
    0.5009467005729675

```python
print([p for p in model.parameters()])
```

    [Parameter containing:
    tensor([[2.5418, 2.0760],
            [1.6006, 2.4416]], requires_grad=True), Parameter containing:
    tensor([6.1531, 4.8214], requires_grad=True), Parameter containing:
    tensor([[-0.8247, -0.1081]], requires_grad=True), Parameter containing:
    tensor([-0.4510], requires_grad=True)]

## Lab
* Get an AWS account and ask for GPU access.
* Add a dataset and data loader to the neural network training.
* Modify the neural network example to solve a regression problem.

```python

```

# References
* https://pytorch.org/docs/stable/index.html
* http://pytorch.org/tutorials/beginner/pytorch_with_examples.html
* https://hsaghir.github.io/data_science/pytorch_starter/


# Non Linear Regression Analysis

Estimated time needed: **20** minutes

## Objectives

After completing this lab you will be able to:

*   Differentiate between linear and non-linear regression
*   Use non-linear regression model in Python

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression since linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example in python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Although linear regression can do a great job at modeling some datasets, it cannot be used for all datasets. First recall how linear regression, models a dataset. It models the linear relationship between a dependent variable y and the independent variables x. It has a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

Non-linear regression is a method to model the non-linear relationship between the independent variables $x$ and the dependent variable $y$. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).  For example:

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and so on. For example: $$ y = \log(x)$$

We can have a function that's even more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying the logarithmic map from the input $x$ to the output $y$. It is one of the simplest form of **log()**: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be a polynomial representation of the $x$ values. In general form it would be written as\
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%202/data/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

**Did you know?** When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset

This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerates slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{-\beta\_1(X-\beta\_2)}}$$

$\beta\_1$: Controls the curve's steepness,

$\beta\_2$: Slides the curve on the x-axis.

### Building The Model

Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?

we can use **curve_fit** which uses non-linear least squares to fit our sigmoid function, to data. Optimize values for the parameters so that the sum of the squared residuals of sigmoid(xdata, \*popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.451712, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice

Can you calculate what is the accuracy of our model?

```python
# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(test_y,y_hat) )

```

    Mean absolute error: 0.03
    Residual sum of squares (MSE): 0.00
    R2-score: 0.98

Click here for the solution

```python
# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(test_y,y_hat) )

```

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

### Thank you for completing this lab!

## Author

Saeed Aghabozorgi

### Other Contributors

Joseph Santarcangelo

## Change Log

| Date (YYYY-MM-DD) | Version | Changed By | Change Description                 |
| ----------------- | ------- | ---------- | ---------------------------------- |
| 2020-11-03        | 2.1     | Lakshmi    | Made changes in URL                |
| 2020-08-27        | 2.0     | Lavanya    | Moved lab to course repo in GitLab |
|                   |         |            |                                    |
|                   |         |            |                                    |

##
 © IBM Corporation 2020. All rights reserved.