#  CS109A  Introduction to Data Science

## Lab 3: plotting, K-NN Regression, Simple Linear Regression

**Harvard University**

**Fall 2019**

**Instructors:** Pavlos Protopapas, Kevin Rader, and Chris Tanner

**Material prepared by**: David Sondak, Will Claybaugh, Pavlos Protopapas, and Eleni Kaxiras.

---

```python
#RUN THIS CELL
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

## Learning Goals

By the end of this lab, you should be able to:
* Review `numpy` including 2-D arrays and understand array reshaping
* Use `matplotlib` to make plots
* Feel comfortable with simple linear regression
* Feel comfortable with $k$ nearest neighbors

**This lab corresponds to lectures 4 and 5 and maps on to homework 2 and beyond.**

## Table of Contents

####  HIGHLIGHTS FROM PRE-LAB

* [1 - Review of numpy](#first-bullet)
* [2 - Intro to matplotlib plus more ](#second-bullet)

####  LAB 3 MATERIAL

* [3 - Simple Linear Regression](#third-bullet)
* [4 - Building a model with `statsmodels` and `sklearn`](#fourth-bullet)
* [5 - Example: Simple linear regression with automobile data](#fifth-bullet)
* [6 - $k$Nearest Neighbors](#sixth-bullet)

```python
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
import time
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)
#import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
# Displays the plots for us.
%matplotlib inline
```

```python
# Use this as a variable to load solutions: %load PATHTOSOLUTIONS/exercise1.py. It will be substituted in the code
# so do not worry if it disappears after you run the cell.
PATHTOSOLUTIONS = 'solutions'
```

## 1 - Review of  the  `numpy` Python library

In lab1 we learned about the `numpy` library [(documentation)](http://www.numpy.org/) and its fast array structure, called the `numpy array`.

```python
# import numpy
import numpy as np
```

```python
# make an array
my_array = np.array([1,4,9,16])
my_array
```

    array([ 1,  4,  9, 16])

```python
print(f'Size of my array: {my_array.size}, or length of my array: {len(my_array)}')
print (f'Shape of my array: {my_array.shape}')
```

    Size of my array: 4, or length of my array: 4
    Shape of my array: (4,)

#### Notice the way the shape appears in numpy arrays

- For a 1D array, .shape returns a tuple with 1 element (n,)
- For a 2D array, .shape returns a tuple with 2 elements (n,m)
- For a 3D array, .shape returns a tuple with 3 elements (n,m,p)

```python
# How to reshape a 1D array to a 2D
my_array.reshape(-1,2)
```

    array([[ 1,  4],
           [ 9, 16]])

Numpy arrays support the same operations as lists! Below we slice and iterate.

```python
print("array[2:4]:", my_array[2:4]) # A slice of the array

# Iterate over the array
for ele in my_array:
    print("element:", ele)
```

    array[2:4]: [ 9 16]
    element: 1
    element: 4
    element: 9
    element: 16

Remember `numpy` gains a lot of its efficiency from being **strongly typed** (all elements are of the same type, such as integer or floating point). If the elements of an array are of a different type, `numpy` will force them into the same type (the longest in terms of bytes)

```python
mixed = np.array([1, 2.3, 'eleni', True])
print(type(1), type(2.3), type('eleni'), type(True))
mixed # all elements will become strings
```

    <class 'int'> <class 'float'> <class 'str'> <class 'bool'>

    array(['1', '2.3', 'eleni', 'True'], dtype='<U32')

Next, we push ahead to two-dimensional arrays and begin to dive into some of the deeper aspects of `numpy`.

```python
# create a 2d-array by handing a list of lists
my_array2d = np.array([ [1, 2, 3, 4],
                        [5, 6, 7, 8],
                        [9, 10, 11, 12]
])

my_array2d
```

    array([[ 1,  2,  3,  4],
           [ 5,  6,  7,  8],
           [ 9, 10, 11, 12]])

### Array Slicing (a reminder...)

Numpy arrays can be sliced, and can be iterated over with loops.  Below is a schematic illustrating slicing two-dimensional arrays.

Notice that the list slicing syntax still works!
`array[2:,3]` says "in the array, get rows 2 through the end, column 3]"
`array[3,:]` says "in the array, get row 3, all columns".

### Pandas Slicing (a reminder...)

`.iloc` is by position (position is unique), `.loc` is by label (label is not unique)

```python
# import cast dataframe
cast = pd.read_csv('../data/cast.csv', encoding='utf_8')
cast.head()
```

      title
      year
      name
      type
      character
      n

      0
      Closet Monster
      2015
      Buffy #1
      actor
      Buffy 4
      31.0

      1
      Suuri illusioni
      1985
      Homo $
      actor
      Guests
      22.0

      2
      Battle of the Sexes
      2017
      $hutter
      actor
      Bobby Riggs Fan
      10.0

      3
      Secret in Their Eyes
      2015
      $hutter
      actor
      2002 Dodger Fan
      NaN

      4
      Steve Jobs
      2015
      $hutter
      actor
      1988 Opera House Patron
      NaN

```python
# get me rows 10 to 13 (python slicing style : exclusive of end)
cast.iloc[10:13]
```

      title
      year
      name
      type
      character
      n

      10
      When the Man Went South
      2014
      Taipaleti 'Atu'ake
      actor
      Two Palms - Ua'i Paame
      8.0

      11
      Little Angel (Angelita)
      2015
      Michael 'babeepower' Viera
      actor
      Chico
      9.0

      12
      Mixing Nia
      1998
      Michael 'babeepower' Viera
      actor
      Rapper
      NaN

```python
# get me columns 0 to 2 but all rows - use head()
cast.iloc[:, 0:2].head()
```

      title
      year

      0
      Closet Monster
      2015

      1
      Suuri illusioni
      1985

      2
      Battle of the Sexes
      2017

      3
      Secret in Their Eyes
      2015

      4
      Steve Jobs
      2015

```python
# get me rows 10 to 13 AND only columns 0 to 2
cast.iloc[10:13, 0:2]
```

      title
      year

      10
      When the Man Went South
      2014

      11
      Little Angel (Angelita)
      2015

      12
      Mixing Nia
      1998

```python
# COMPARE: get me rows 10 to 13 (pandas slicing style : inclusive of end)
cast.loc[10:13]
```

      title
      year
      name
      type
      character
      n

      10
      When the Man Went South
      2014
      Taipaleti 'Atu'ake
      actor
      Two Palms - Ua'i Paame
      8.0

      11
      Little Angel (Angelita)
      2015
      Michael 'babeepower' Viera
      actor
      Chico
      9.0

      12
      Mixing Nia
      1998
      Michael 'babeepower' Viera
      actor
      Rapper
      NaN

      13
      The Replacements
      2000
      Steven 'Bear'Boyd
      actor
      Defensive Tackle - Washington Sentinels
      NaN

```python
# give me columns 'year' and 'type' by label but only for rows 5 to 10
cast.loc[5:10,['year','type']]
```

      year
      type

      5
      2015
      actor

      6
      2015
      actor

      7
      2009
      actor

      8
      2014
      actor

      9
      2014
      actor

      10
      2014
      actor

```python
# cast.iloc[5:10,['year','type']] # errors out
cast.iloc[5:10,[1, 3]]
```

      year
      type

      5
      2015
      actor

      6
      2015
      actor

      7
      2009
      actor

      8
      2014
      actor

      9
      2014
      actor

### Python Trick of the Day

```python
import re
names = ['mayday','springday','horseday','june']
```

```python
# TODO : substitute these lines code with 1 line of code using list comprehension

cleaned = []
for name in names:
    this = re.sub('[Dd]ay$', '', name)
    cleaned.append(this)
cleaned
```

    ['may', 'spring', 'horse', 'june']

```python
# your code here
[re.sub('[dD]ay', '', name) for name in names]
```

    ['may', 'spring', 'horse', 'june']

```python
# solution
cleaned2 = [re.sub('[Dd]ay$', '', name) for name in names]
cleaned2
```

    ['may', 'spring', 'horse', 'june']

## 2 - Plotting with matplotlib and beyond

`matplotlib` is a very powerful `python` library for making scientific plots.

We will not focus too much on the internal aspects of `matplotlib` in today's lab. There are many excellent tutorials out there for `matplotlib`.  For example,
* [`matplotlib` homepage](https://matplotlib.org/)
* [`matplotlib` tutorial](https://github.com/matplotlib/AnatomyOfMatplotlib)

Conveying your findings convincingly is an absolutely crucial part of any analysis. Therefore, you must be able to write well and make compelling visuals.  Creating informative visuals is an involved process and we won't cover that in this lab.  However, part of creating informative data visualizations means generating *readable* figures.  If people can't read your figures or have a difficult time interpreting them, they won't understand the results of your work.  Here are some non-negotiable commandments for any plot:
* Label $x$ and $y$ axes
* Axes labels should be informative
* Axes labels should be large enough to read
* Make tick labels large enough
* Include a legend if necessary
* Include a title if necessary
* Use appropriate line widths
* Use different line styles for different lines on the plot
* Use different markers for different lines

There are other important elements, but that list should get you started on your way.

We will work with `matplotlib` and `seaborn` for plotting in this class.  `matplotlib` is a very powerful `python` library for making scientific plots.  `seaborn` is a little more specialized in that it was developed for statistical data visualization.  We will cover some `seaborn` later in class. In the meantime you can look at the [seaborn documentation](https://seaborn.pydata.org)

First, let's generate some data.

#### Let's plot some functions

We will use the following three functions to make some plots:

* Logistic function:
  \begin{align*}
    f\left(z\right) = \dfrac{1}{1 + be^{-az}}
  \end{align*}
  where $a$ and $b$ are parameters.
* Hyperbolic tangent:
  \begin{align*}
    g\left(z\right) = b\tanh\left(az\right) + c
  \end{align*}
  where $a$, $b$, and $c$ are parameters.
* Rectified Linear Unit:
  \begin{align*}
    h\left(z\right) =
    \left\{
      \begin{array}{lr}
        z, \quad z > 0 \\
        \epsilon z, \quad z\leq 0
      \end{array}
    \right.
  \end{align*}
  where $\epsilon < 0$ is a small, positive parameter.

You are given the code for the first two functions.  Notice that $z$ is passed in as a `numpy` array and that the functions are returned as `numpy` arrays.  Parameters are passed in as floats.

You should write a function to compute the rectified linear unit.  The input should be a `numpy` array for $z$ and a positive float for $\epsilon$.

```python
import numpy as np

def logistic(z: np.ndarray, a: float, b: float) -> np.ndarray:
    """ Compute logistic function
      Inputs:
         a: exponential parameter
         b: exponential prefactor
         z: numpy array; domain
      Outputs:
         f: numpy array of floats, logistic function
    """

    den = 1.0 + b * np.exp(-a * z)
    return 1.0 / den

def stretch_tanh(z: np.ndarray, a: float, b: float, c: float) -> np.ndarray:
    """ Compute stretched hyperbolic tangent
      Inputs:
         a: horizontal stretch parameter (a>1 implies a horizontal squish)
         b: vertical stretch parameter
         c: vertical shift parameter
         z: numpy array; domain
      Outputs:
         g: numpy array of floats, stretched tanh
    """
    return b * np.tanh(a * z) + c

def relu(z: np.ndarray, eps: float = 0.01) -> np.ndarray:
    """ Compute rectificed linear unit
      Inputs:
         eps: small positive parameter
         z: numpy array; domain
      Outputs:
         h: numpy array; relu
    """
    return np.fmax(z, eps * z)
```

Now let's make some plots.  First, let's just warm up and plot the logistic function.

```python
x = np.linspace(-5.0, 5.0, 100) # Equally spaced grid of 100 pts between -5 and 5

f = logistic(x, 1.0, 1.0) # Generate data
```

```python
plt.plot(x, f)
plt.xlabel('x')
plt.ylabel('f')
plt.title('Logistic Function')
plt.grid(True)
```

#### Figures with subplots

Let's start thinking about the plots as objects. We have the `figure` object which is like a matrix of smaller plots named `axes`. You can use array notation when handling it.

```python
fig, ax = plt.subplots(1,1) # Get figure and axes objects

ax.plot(x, f) # Make a plot

# Create some labels
ax.set_xlabel('x')
ax.set_ylabel('f')
ax.set_title('Logistic Function')

# Grid
ax.grid(True)
```

Wow, it's *exactly* the same plot!  Notice, however, the use of `ax.set_xlabel()` instead of `plt.xlabel()`.  The difference is tiny, but you should be aware of it.  I will use this plotting syntax from now on.

What else do we need to do to make this figure better?  Here are some options:
* Make labels bigger!
* Make line fatter
* Make tick mark labels bigger
* Make the grid less pronounced
* Make figure bigger

Let's get to it.

```python
fig, ax = plt.subplots(1,1, figsize=(10,6)) # Make figure bigger

# Make line plot
ax.plot(x, f, lw=4)

# Update ticklabel size
ax.tick_params(labelsize=24)

# Make labels
ax.set_xlabel(r'$x$', fontsize=24) # Use TeX for mathematical rendering
ax.set_ylabel(r'$f(x)$', fontsize=24) # Use TeX for mathematical rendering
ax.set_title('Logistic Function', fontsize=24)

ax.grid(True, lw=1.5, ls='--', alpha=0.75)
```

Notice:
* `lw` stands for `linewidth`.  We could also write `ax.plot(x, f, linewidth=4)`
* `ls` stands for `linestyle`.
* `alpha` stands for transparency.

The only thing remaining to do is to change the $x$ limits.  Clearly these should go from $-5$ to $5$.

```python
#fig.savefig('logistic.png')

# Put this in a markdown cell and uncomment this to check what you saved.
#
```

#### Resources
If you want to see all the styles available, please take a look at the documentation.
* [Line styles](https://matplotlib.org/2.0.1/api/lines_api.html#matplotlib.lines.Line2D.set_linestyle)
* [Marker styles](https://matplotlib.org/2.0.1/api/markers_api.html#module-matplotlib.markers)
* [Everything you could ever want](https://matplotlib.org/2.0.1/api/lines_api.html#matplotlib.lines.Line2D.set_marker)

We haven't discussed it yet, but you can also put a legend on a figure.  You'll do that in the next exercise.  Here are some additional resources:
* [Legend](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.legend.html)
* [Grid](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.grid.html)

`ax.legend(loc='best', fontsize=24);`

Exercise

Do the following:
* Make a figure with the logistic function, hyperbolic tangent, and rectified linear unit.
* Use different line styles for each plot
* Put a legend on your figure

Here's an example of a figure:

```python
# your code here

# First get the data
f = logistic(x, 2.0, 1.0)
g = stretch_tanh(x, 2.0, 0.5, 0.5)
h = relu(x)

fig, ax = plt.subplots(1,1, figsize=(10,6)) # Create figure object

# Make actual plots
# (Notice the label argument!)
ax.plot(x, f, lw=4, ls='-', label=r'$L(x;1)$')
ax.plot(x, g, lw=4, ls='--', label=r'$\tanh(2x)$')
ax.plot(x, h, lw=4, ls='-.', label=r'$relu(x; 0.01)$')

# Make the tick labels readable
ax.tick_params(labelsize=24)

# Set axes limits to make the scale nice
ax.set_xlim(x.min(), x.max())
ax.set_ylim(h.min(), 1.1)

# Make readable labels
ax.set_xlabel(r'$x$', fontsize=24)
ax.set_ylabel(r'$h(x)$', fontsize=24)
ax.set_title('Activation Functions', fontsize=24)

# Set up grid
ax.grid(True, lw=1.75, ls='--', alpha=0.75)

# Put legend on figure
ax.legend(loc='best', fontsize=24);

fig.savefig('../images/nice_plots.png')
```

Exercise

These figures look nice in the plot and it makes sense for comparison. Now let's put the 3 different figures in separate plots.

* Make a separate plot for each figure and line them up on the same row.

```python
# your code here

```

```python
# %load solutions/three_subplots.py
# First get the data
f = logistic(x, 2.0, 1.0)
g = stretch_tanh(x, 2.0, 0.5, 0.5)
h = relu(x)

fig, ax = plt.subplots(1,3, figsize=(20,6)) # Create figure object

# Make actual plots
ax[0].plot(x, f, lw=4, ls='-', label=r'$L(x;1)$')
ax[1].plot(x, g, lw=4, ls='--', label=r'$\tanh(2x)$')
ax[2].plot(x, h, lw=4, ls='-.', label=r'$relu(x; 0.01)$')

# Make the tick labels readable
ax[0].tick_params(labelsize=24)
ax[1].tick_params(labelsize=24)
ax[2].tick_params(labelsize=24)

# Set axes limits to make the scale nice
ax[0].set_xlim(x.min(), x.max())
ax[0].set_ylim(h.min(), 1.1)
ax[1].set_xlim(x.min(), x.max())
ax[1].set_ylim(h.min(), 1.1)
ax[2].set_xlim(x.min(), x.max())
ax[2].set_ylim(h.min(), 1.1)

# Make readable labels
ax[0].set_xlabel(r'$x$', fontsize=24)
ax[0].set_ylabel(r'$h(x)$', fontsize=24)
ax[0].set_title('Activation Functions', fontsize=24)

ax[1].set_xlabel(r'$x$', fontsize=24)
ax[1].set_ylabel(r'$h(x)$', fontsize=24)
ax[1].set_title('Activation Functions', fontsize=24)

ax[2].set_xlabel(r'$x$', fontsize=24)
ax[2].set_ylabel(r'$h(x)$', fontsize=24)
ax[2].set_title('Activation Functions', fontsize=24)

# Set up grid
ax[0].grid(True, lw=1.75, ls='--', alpha=0.75)
ax[1].grid(True, lw=1.75, ls='--', alpha=0.75)
ax[2].grid(True, lw=1.75, ls='--', alpha=0.75)

# Put legend on figure
ax[0].legend(loc='best', fontsize=24);
ax[1].legend(loc='best', fontsize=24);
ax[2].legend(loc='best', fontsize=24);

#fig.savefig('../images/nice_sub_plots.png')

```

Exercise

* Make a grid of 2 x 3 separate plots, 3 will be empty. Just plot the functions and do not worry about cosmetics. We just want you ro see the functionality.

```python
# your code here

```

```python
# %load solutions/six_subplots.py

# First get the data
f = logistic(x, 2.0, 1.0)
g = stretch_tanh(x, 2.0, 0.5, 0.5)
h = relu(x)

fig, ax = plt.subplots(2,3, figsize=(20,6)) # Create figure object

# Make actual plots
ax[0][0].plot(x, f, lw=4, ls='-', label=r'$L(x;1)$')
ax[1][1].plot(x, g, lw=4, ls='--', label=r'$\tanh(2x)$')
ax[1][2].plot(x, h, lw=4, ls='-.', label=r'$relu(x; 0.01)$')
ax[0][2].plot(x, h, lw=4, ls='-.', label=r'$relu(x; 0.01)$')

```

## 3 - Simple Linear Regression

Linear regression and its many extensions are a workhorse of the statistics and data science community, both in application and as a reference point for other models. Most of the major concepts in machine learning can be and often are discussed in terms of various linear regression models. Thus, this section will introduce you to building and fitting linear regression models and some of the process behind it, so that you can 1) fit models to data you encounter 2) experiment with different kinds of linear regression and observe their effects 3) see some of the technology that makes regression models work.

### Linear regression with a toy dataset
We first examine a toy problem, focusing our efforts on fitting a linear model to a small dataset with three observations.  Each observation consists of one predictor $x_i$ and one response $y_i$ for $i = 1, 2, 3$,

\begin{align*}
(x , y) = \{(x_1, y_1), (x_2, y_2), (x_3, y_3)\}.
\end{align*}

To be very concrete, let's set the values of the predictors and responses.

\begin{equation*}
(x , y) = \{(1, 2), (2, 2), (3, 4)\}
\end{equation*}

There is no line of the form $\beta_0 + \beta_1 x = y$ that passes through all three observations, since the data are not collinear. Thus our aim is to find the line that best fits these observations in the *least-squares sense*, as discussed in lecture.

Exercise (for home)

* Make two numpy arrays out of this data, x_train and y_train
* Check the dimentions of these arrays
* Try to reshape them into a different shape
* Make points into a very simple scatterplot
* Make a better scatterplot

```python
# your code here
x_train = np.array([1, 2, 3]).reshape(3, 1)
y_train = np.array([2, 2, 4]).reshape(3, 1)
print('x shape: {}, y shape: {}'.format(x_train.shape, y_train.shape))
fig, ax = plt.subplots(1, 1, figsize=(5, 5))
ax.scatter(x_train, y_train)
```

```python
# solution
x_train = np.array([1,2,3])
y_train = np.array([2,3,6])
type(x_train)
```

    numpy.ndarray

```python
x_train.shape
```

    (3,)

```python
x_train = x_train.reshape(3,1)
x_train.shape
```

    (3, 1)

```python
# %load solutions/simple_scatterplot.py
# Make a simple scatterplot
plt.scatter(x_train,y_train)

# check dimensions
print(x_train.shape,y_train.shape)

```

```python
# %load solutions/nice_scatterplot.py
def nice_scatterplot(x, y, title):
    # font size
    f_size = 18

    # make the figure
    fig, ax = plt.subplots(1,1, figsize=(8,5)) # Create figure object

    # set axes limits to make the scale nice
    ax.set_xlim(np.min(x)-1, np.max(x) + 1)
    ax.set_ylim(np.min(y)-1, np.max(y) + 1)

    # adjust size of tickmarks in axes
    ax.tick_params(labelsize = f_size)

    # remove tick labels
    ax.tick_params(labelbottom=False,  bottom=False)

    # adjust size of axis label
    ax.set_xlabel(r'$x$', fontsize = f_size)
    ax.set_ylabel(r'$y$', fontsize = f_size)

    # set figure title label
    ax.set_title(title, fontsize = f_size)

    # you may set up grid with this
    ax.grid(True, lw=1.75, ls='--', alpha=0.15)

    # make actual plot (Notice the label argument!)
    #ax.scatter(x, y, label=r'$my points$')
    #ax.scatter(x, y, label='$my points$')
    ax.scatter(x, y, label=r'$my\,points$')
    ax.legend(loc='best', fontsize = f_size);

    return ax

nice_scatterplot(x_train, y_train, 'hello nice plot')

```

#### Formulae
Linear regression is special among the models we study because it can be solved explicitly. While most other models (and even some advanced versions of linear regression) must be solved itteratively, linear regression has a formula where you can simply plug in the data.

For the single predictor case it is:
    \begin{align}
      \beta_1 &= \frac{\sum_{i=1}^n{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^n{(x_i-\bar{x})^2}}\\
      \beta_0 &= \bar{y} - \beta_1\bar{x}\
    \end{align}

Where $\bar{y}$ and $\bar{x}$ are the mean of the y values and the mean of the x values, respectively.

###  Building a model from scratch
In this part, we will solve the equations for simple linear regression and find the best fit solution to our toy problem.

The snippets of code below implement the linear regression equations on the observed predictors and responses, which we'll call the training data set.  Let's walk through the code.

We have to reshape our arrrays to 2D. We will see later why.

Exercise

* make an array with shape (2,3)
* reshape it to a size that you want

```python
# your code here

```

```python
#solution
xx = np.array([[1,2,3],[4,6,8]])
xxx = xx.reshape(-1,2)
xxx.shape
```

    (3, 2)

```python
# Reshape to be a proper 2D array
x_train = x_train.reshape(x_train.shape[0], 1)
y_train = y_train.reshape(y_train.shape[0], 1)

print(x_train.shape)
```

    (3, 1)

```python
# first, compute means
y_bar = np.mean(y_train)
x_bar = np.mean(x_train)

# build the two terms
numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
denominator = np.sum((x_train - x_bar)**2)

print(numerator.shape, denominator.shape) #check shapes
```

    () ()

* Why the empty brackets? (The numerator and denominator are scalars, as expected.)

```python
#slope beta1
beta_1 = numerator/denominator

#intercept beta0
beta_0 = y_bar - beta_1*x_bar

print("The best-fit line is {0:4.2f} + {1:3.3f} * x".format(beta_0, beta_1))
print(f'The best fit is {beta_0}')
```

    The best-fit line is -0.33 + 2.000 * x
    The best fit is -0.3333333333333335

Exercise

Turn the code from the above cells into a function called `simple_linear_regression_fit`, that inputs the training data and returns `beta0` and `beta1`.

To do this, copy and paste the code from the above cells below and adjust the code as needed, so that the training data becomes the input and the betas become the output.

```python
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:

    return
```

Check your function by calling it with the training data from above and printing out the beta values.

```python
# Your code here
def my_simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:
    x_bar = np.mean(x_train)
    y_bar = np.mean(y_train)
    x_train = x_train.reshape(-1, 1)
    y_train = y_train.reshape(-1, 1)
    num = np.sum( (x_train - x_bar) * (y_train - y_bar) )
    den = np.sum( (x_train - x_bar)** 2)
    beta1 = num/den
    beta0 = y_bar - (beta1 * x_bar)
    return np.array([beta0, beta1])
simple_linear_regression_fit(x_train, y_train)
```

    array([-0.33333333,  2.        ])

```python
# %load solutions/simple_linear_regression_fit.py
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:
    """
    Inputs:
    x_train: a (num observations by 1) array holding the values of the predictor variable
    y_train: a (num observations by 1) array holding the values of the response variable

    Returns:
    beta_vals:  a (num_features by 1) array holding the intercept and slope coeficients
    """

    # Check input array sizes
    if len(x_train.shape) < 2:
        print("Reshaping features array.")
        x_train = x_train.reshape(x_train.shape[0], 1)

    if len(y_train.shape) < 2:
        print("Reshaping observations array.")
        y_train = y_train.reshape(y_train.shape[0], 1)

    # first, compute means
    y_bar = np.mean(y_train)
    x_bar = np.mean(x_train)

    # build the two terms
    numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
    denominator = np.sum((x_train - x_bar)**2)

    #slope beta1
    beta_1 = numerator/denominator

    #intercept beta0
    beta_0 = y_bar - beta_1*x_bar

    return np.array([beta_0,beta_1])
simple_linear_regression_fit(x_train, y_train)
```

    array([-0.33333333,  2.        ])

* Let's run this function and see the coefficients

```python
x_train = np.array([1 ,2, 3])
y_train = np.array([2, 2, 4])

betas = simple_linear_regression_fit(x_train, y_train)

beta_0 = betas[0]
beta_1 = betas[1]

print("The best-fit line is {0:8.6f} + {1:8.6f} * x".format(beta_0, beta_1))

betas = my_simple_linear_regression_fit(x_train, y_train)

beta_0 = betas[0]
beta_1 = betas[1]

print("The best-fit line is {0:8.6f} + {1:8.6f} * x".format(beta_0, beta_1))
```

    Reshaping features array.
    Reshaping observations array.
    The best-fit line is 0.666667 + 1.000000 * x
    The best-fit line is 0.666667 + 1.000000 * x

Exercise

* Do the values of `beta0` and `beta1` seem reasonable?
* Plot the training data using a scatter plot.
* Plot the best fit line with `beta0` and `beta1` together with the training data.

```python
# Your code here
```

```python
# %load solutions/best_fit_scatterplot.py
fig_scat, ax_scat = plt.subplots(1,1, figsize=(10,6))

# Plot best-fit line
x_train = np.array([[1, 2, 3]]).T

best_fit = beta_0 + beta_1 * x_train

ax_scat.scatter(x_train, y_train, s=300, label='Training Data')
ax_scat.plot(x_train, best_fit, ls='--', label='Best Fit Line')

ax_scat.set_xlabel(r'$x_{train}$')
ax_scat.set_ylabel(r'$y$');

```

The values of `beta0` and `beta1` seem roughly reasonable.  They capture the positive correlation.  The line does appear to be trying to get as close as possible to all the points.

## 4 - Building a model with `statsmodels` and `sklearn`

Now that we can concretely fit the training data from scratch, let's learn two `python` packages to do it all for us:
* [statsmodels](http://www.statsmodels.org/stable/regression.html) and
* [scikit-learn (sklearn)](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

Our goal  is to show how to implement simple linear regression with these packages.  For an important sanity check, we compare the $\beta$ values from `statsmodels` and `sklearn` to the $\beta$ values that we found from above with our own implementation.

For the purposes of this lab, `statsmodels` and `sklearn` do the same thing.  More generally though, `statsmodels` tends to be easier for inference \[finding the values of the slope and intercept and dicussing uncertainty in those values\], whereas `sklearn` has machine-learning algorithms and is better for prediction \[guessing y values for a given x value\]. (Note that both packages make the same guesses, it's just a question of which activity they provide more support for.

**Note:** `statsmodels` and `sklearn` are different packages!  Unless we specify otherwise, you can use either one.

Below is the code for `statsmodels`.  `Statsmodels` does not by default include the column of ones in the $X$ matrix, so we include it manually with `sm.add_constant`.

```python
import statsmodels.api as sm
```

```python
print(x_train)
```

    [1 2 3]

```python
# create the X matrix by appending a column of ones to x_train
X = sm.add_constant(x_train)

# this is the same matrix as in our scratch problem!
print(X)

# build the OLS model (ordinary least squares) from the training data
toyregr_sm = sm.OLS(y_train, X)

# do the fit and save regression info (parameters, etc) in results_sm
results_sm = toyregr_sm.fit()

# pull the beta parameters out from results_sm
beta0_sm = results_sm.params[0]
beta1_sm = results_sm.params[1]

print(f'The regression coef from statsmodels are: beta_0 = {beta0_sm:8.6f} and beta_1 = {beta1_sm:8.6f}')
```

    [[1. 1.]
     [1. 2.]
     [1. 3.]]
    The regression coef from statsmodels are: beta_0 = 0.666667 and beta_1 = 1.000000

Besides the beta parameters, `results_sm` contains a ton of other potentially useful information.

```python
import warnings
warnings.filterwarnings('ignore')
print(results_sm.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.750
    Model:                            OLS   Adj. R-squared:                  0.500
    Method:                 Least Squares   F-statistic:                     3.000
    Date:                Thu, 26 Nov 2020   Prob (F-statistic):              0.333
    Time:                        11:15:25   Log-Likelihood:                -2.0007
    No. Observations:                   3   AIC:                             8.001
    Df Residuals:                       1   BIC:                             6.199
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.6667      1.247      0.535      0.687     -15.181      16.514
    x1             1.0000      0.577      1.732      0.333      -6.336       8.336
    ==============================================================================
    Omnibus:                          nan   Durbin-Watson:                   3.000
    Prob(Omnibus):                    nan   Jarque-Bera (JB):                0.531
    Skew:                          -0.707   Prob(JB):                        0.767
    Kurtosis:                       1.500   Cond. No.                         6.79
    ==============================================================================

    Notes:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Now let's turn our attention to the `sklearn` library.

```python
from sklearn import linear_model
```

```python
# build the least squares model
toyregr = linear_model.LinearRegression()

# save regression info (parameters, etc) in results_skl
results = toyregr.fit(x_train, y_train)

# pull the beta parameters out from results_skl
beta0_skl = toyregr.intercept_
beta1_skl = toyregr.coef_[0]

print("The regression coefficients from the sklearn package are: beta_0 = {0:8.6f} and beta_1 = {1:8.6f}".format(beta0_skl, beta1_skl))
```

    The regression coefficients from the sklearn package are: beta_0 = 0.666667 and beta_1 = 1.000000

We should feel pretty good about ourselves now, and we're ready to move on to a real problem!

### The `scikit-learn` library and the shape of things

Before diving into a "real" problem, let's discuss more of the details of `sklearn`.

`Scikit-learn` is the main `Python` machine learning library. It consists of many learners which can learn models from data, as well as a lot of utility functions such as `train_test_split()`.

Use the following to add the library into your code:

```python
import sklearn
```

In `scikit-learn`, an **estimator** is a Python object that implements the methods `fit(X, y)` and `predict(T)`

Let's see the structure of `scikit-learn` needed to make these fits. `fit()` always takes two arguments:
```python
estimator.fit(Xtrain, ytrain)
```
We will consider two estimators in this lab: `LinearRegression` and `KNeighborsRegressor`.

It is very important to understand that `Xtrain` must be in the form of a **2x2 array** with each row corresponding to one sample, and each column corresponding to the feature values for that sample.

`ytrain` on the other hand is a simple array of responses.  These are continuous for regression problems.

### Practice with `sklearn` and a real dataset
We begin by loading up the `mtcars` dataset. This data was extracted from the 1974 Motor Trend US magazine, and comprises of fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models). We will load this data to a dataframe with 32 observations on 11 (numeric) variables. Here is an explanation of the features:

- `mpg` is Miles/(US) gallon
- `cyl` is Number of cylinders,
- `disp` is	Displacement (cu.in.),
- `hp` is	Gross horsepower,
- `drat` is	Rear axle ratio,
- `wt` is the Weight (1000 lbs),
- `qsec` is 1/4 mile time,
- `vs` is Engine (0 = V-shaped, 1 = straight),
- `am` is Transmission (0 = automatic, 1 = manual),
- `gear` is the Number of forward gears,
- `carb` is	Number of carburetors.

```python
import pandas as pd

#load mtcars
dfcars = pd.read_csv("../data/mtcars.csv")
dfcars.head()
```

      Unnamed: 0
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
# Fix the column title
dfcars = dfcars.rename(columns={"Unnamed: 0":"car name"})
dfcars.head()
```

      car name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
dfcars.shape
```

    (32, 12)

#### Searching for values: how many cars have 4 gears?

```python
(dfcars[dfcars.gear == 4].gear).count()
```

    12

```python
len(dfcars[dfcars.gear == 4].drop_duplicates(subset='car name', keep='first'))
```

    12

Next, let's split the dataset into a training set and test set.

```python
# split into training set and testing set
from sklearn.model_selection import train_test_split

#set random_state to get the same split every time
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)
```

```python
# testing set is around 20% of the total data; training set is around 80%
print("Shape of full dataset is: {0}".format(dfcars.shape))
print("Shape of training dataset is: {0}".format(traindf.shape))
print("Shape of test dataset is: {0}".format(testdf.shape))
```

    Shape of full dataset is: (32, 12)
    Shape of training dataset is: (25, 12)
    Shape of test dataset is: (7, 12)

Now we have training and test data.  We still need to select a predictor and a response from this dataset.  Keep in mind that we need to choose the predictor and response from both the training and test set.  You will do this in the exercises below.  However, we provide some starter code for you to get things going.

```python
traindf.head()
```

      car name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      25
      Fiat X1-9
      27.3
      4
      79.0
      66
      4.08
      1.935
      18.90
      1
      1
      4
      1

      12
      Merc 450SL
      17.3
      8
      275.8
      180
      3.07
      3.730
      17.60
      0
      0
      3
      3

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

      16
      Chrysler Imperial
      14.7
      8
      440.0
      230
      3.23
      5.345
      17.42
      0
      0
      3
      4

```python
# Extract the response variable that we're interested in
y_train = traindf.mpg
y_train
```

    25    27.3
    12    17.3
    0     21.0
    4     18.7
    16    14.7
    5     18.1
    13    15.2
    11    16.4
    23    13.3
    1     21.0
    2     22.8
    26    26.0
    3     21.4
    21    15.5
    27    30.4
    22    15.2
    18    30.4
    31    21.4
    20    21.5
    7     24.4
    10    17.8
    14    10.4
    28    15.8
    19    33.9
    6     14.3
    Name: mpg, dtype: float64

Exercise

Use slicing to get the same vector `y_train`

----

```python
y_train_new = traindf.iloc[:, 1]
(y_train_new == y_train).all()
```

    True

```python
y_train_new is y_train
```

    False

Now, notice the shape of `y_train`.

```python
y_train.shape, type(y_train)
```

    ((25,), pandas.core.series.Series)

### Array reshape
This is a 1D array as should be the case with the **Y** array.  Remember, `sklearn` requires a 2D array only for the predictor array.  You will have to pay close attention to this in the exercises later. `Sklearn` doesn't care too much about the shape of `y_train`.

The whole reason we went through that whole process was to show you how to reshape your data into the correct format.

**IMPORTANT:** Remember that your response variable `ytrain` can be a vector but your predictor variable `xtrain` ***must*** be an array!

## 5 - Example: Simple linear regression with automobile data
We will now use `sklearn` to predict automobile mileage per gallon (mpg) and evaluate these predictions. We already loaded the data and split them into a training set and a test set.

We need to choose the variables that we think will be good predictors for the dependent variable `mpg`.

Exercise in pairs

* Pick one variable to use as a predictor for simple linear regression.  Discuss your reasons with the person next to you.
* Justify your choice with some visualizations.
* Is there a second variable you'd like to use? For example, we're not doing multiple linear regression here, but if we were, is there another variable you'd like to include if we were using two predictors?

```python
traindf.columns
```

    Index(['car name', 'mpg', 'cyl', 'disp', 'hp', 'drat', 'wt', 'qsec', 'vs', 'am', 'gear', 'carb'], dtype='object')

```python
x_wt = dfcars.wt
x_wt.shape
```

    (32,)

```python
# Your code here
# I am choosing qsec as my prominent predictor variable as it seems to
# capture the effects of hp, cyl, wt etc.

from sklearn.linear_model import LinearRegression
lr = LinearRegression()

x_qsec_train = traindf.qsec.values
y_train = traindf.mpg

# fit the model and predict
lr.fit(x_qsec_train.reshape(-1, 1), y_train)
y_train_pred = lr.predict(x_qsec_train.reshape(-1, 1))

# let's plot the actual and predicted mpg values
plt.scatter(x_qsec_train, y_train, label='truth')
plt.plot(x_qsec_train, y_train_pred, 'r-', label='prediction')
plt.xlabel('qsec (in seconds)')
plt.ylabel('mpg (in miles/gallon)')
plt.legend()
plt.show();
```

```python
# %load solutions/cars_simple_EDA.py
y_mpg = dfcars.mpg
x_wt = dfcars.wt
x_hp = dfcars.hp

fig_wt, ax_wt = plt.subplots(1,1, figsize=(10,6))
ax_wt.scatter(x_wt, y_mpg)
ax_wt.set_xlabel(r'Car Weight')
ax_wt.set_ylabel(r'Car MPG')

fig_hp, ax_hp = plt.subplots(1,1, figsize=(10,6))
ax_hp.scatter(x_hp, y_mpg)
ax_hp.set_xlabel(r'Car HP')
ax_hp.set_ylabel(r'Car MPG')

```

Exercise

* Use `sklearn` to fit the training data using simple linear regression.
* Use the model to make mpg predictions on the test set.
* Plot the data and the prediction.
* Print out the mean squared error for the training set and the test set and compare.

```python
from sklearn.metrics import mean_squared_error
x_qsec_train, y_train = traindf.qsec.values, traindf.mpg
x_qsec_test, y_test = testdf.qsec.values, testdf.mpg

# fit the model and predict
lr.fit(x_qsec_train.reshape(-1, 1), y_train)
y_train_pred = lr.predict(x_qsec_train.reshape(-1, 1))
y_test_pred = lr.predict(x_qsec_test.reshape(-1, 1))

# let's plot the actual and predicted mpg values on train data
fig, ax = plt.subplots(1, 2, figsize=(16, 6))
ax[0].scatter(x_qsec_train, y_train, label='truth')
ax[0].plot(x_qsec_train, y_train_pred, 'r-', label='prediction (MSE: {0:4.2f})'.format(mean_squared_error(y_train, y_train_pred)))
ax[0].set_xlabel('qsec (in seconds)')
ax[0].set_ylabel('mpg')
ax[0].legend()
ax[0].set_title('LR on training data')

# let's plot the actual and predicted mpg values on test data
ax[1].scatter(x_qsec_test, y_test, label='truth')
ax[1].plot(x_qsec_test, y_test_pred, 'r-', label='prediction (MSE: {0:4.2f})'.format(mean_squared_error(y_test, y_test_pred)))
ax[1].set_xlabel('qsec (in seconds)')
ax[1].set_ylabel('mpg')
ax[1].legend()
ax[1].set_title('LR on test data')

plt.show();
```

```python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

dfcars = pd.read_csv("../data/mtcars.csv")
dfcars = dfcars.rename(columns={"Unnamed: 0":"name"})

dfcars.head()
```

      name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)

y_train = np.array(traindf.mpg)
X_train = np.array(traindf.wt)
X_train = X_train.reshape(X_train.shape[0], 1)
```

```python
y_test = np.array(testdf.mpg)
X_test = np.array(testdf.wt)
X_test = X_test.reshape(X_test.shape[0], 1)
```

```python
# Let's take another look at our data
dfcars.head()
```

      name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
# And out train and test sets
y_train.shape, X_train.shape
```

    ((25,), (25, 1))

```python
y_test.shape, X_test.shape
```

    ((7,), (7, 1))

```python
#create linear model
regression = LinearRegression()

#fit linear model
regression.fit(X_train, y_train)

predicted_y = regression.predict(X_test)

r2 = regression.score(X_test, y_test)
print(f'R^2 = {r2:.5}')
```

    R^2 = 0.68798

```python
print(regression.score(X_train, y_train))

print(mean_squared_error(predicted_y, y_test))
print(mean_squared_error(y_train, regression.predict(X_train)))

print('Coefficients: \n', regression.coef_[0], regression.intercept_)
```

    0.7701379909791617
    12.475985659918837
    7.773697766387512
    Coefficients:
     -5.336941400557081 36.93731031351841

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))
ax.plot(y_test, predicted_y, 'o')
grid = np.linspace(np.min(dfcars.mpg), np.max(dfcars.mpg), 100)
ax.plot(grid, grid, color="black") # 45 degree line
ax.set_xlabel("actual y")
ax.set_ylabel("predicted y")
ax.grid('True', alpha=0.5)

fig1, ax1 = plt.subplots(1,1, figsize=(10,6))
ax1.plot(dfcars.wt, dfcars.mpg, 'o')
xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)
ax1.plot(xgrid, regression.predict(xgrid.reshape(100, 1)))
ax1.grid('True', alpha=0.5)
```

## 6 - $k$-nearest neighbors

Now that you're familiar with `sklearn`, you're ready to do a KNN regression.

Sklearn's regressor is called `sklearn.neighbors.KNeighborsRegressor`. Its main parameter is the `number of nearest neighbors`. There are other parameters such as the distance metric (default for 2 order is the Euclidean distance). For a list of all the parameters see the [Sklearn kNN Regressor Documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html).

Let's use $5$ nearest neighbors.

```python
# Import the library
from sklearn.neighbors import KNeighborsRegressor
```

```python
# Set number of neighbors
k = 5
knnreg = KNeighborsRegressor(n_neighbors=k)
```

```python
# Fit the regressor - make sure your numpy arrays are the right shape
knnreg.fit(X_train, y_train)

# Evaluate the outcome on the train set using R^2
r2_train = knnreg.score(X_train, y_train)

# Print results
print(f'kNN model with {k} neighbors gives R^2 on the train set: {r2_train:.5}')
```

    kNN model with 5 neighbors gives R^2 on the train set: 0.87181

```python
knnreg.predict(X_test)
```

    array([20.14, 14.  , 15.3 , 26.3 , 19.56, 17.06, 16.88])

Exercise

Calculate and print the $R^{2}$ score on the test set

```python
# Your code here
r2_test = knnreg.score(X_test, y_test)
print('KNN model with {} neighbors gives R^2 on the test set: {:.5f}'.format(k, r2_test))
```

    KNN model with 5 neighbors gives R^2 on the test set: 0.69922

```python
# calculate r2 on my own
y_pred = knnreg.predict(X_test)
num = np.sum( (y_pred - y_test)**2 )
den = np.sum( (np.mean(y_test) - y_test)**2 )
r2 = 1 - (num / den)
print(r2)
```

    0.6992188728167332

Not so good? Lets vary the number of neighbors and see what we get.

```python
# Make our lives easy by storing the different regressors in a dictionary
regdict = {}

# Make our lives easier by entering the k values from a list
k_list = [1, 2, 4, 15]

# Do a bunch of KNN regressions
for k in k_list:
    knnreg = KNeighborsRegressor(n_neighbors=k)
    knnreg.fit(X_train, y_train)
    # Store the regressors in a dictionary
    regdict[k] = knnreg

# Print the dictionary to see what we have
regdict
```

    {1: KNeighborsRegressor(n_neighbors=1),
     2: KNeighborsRegressor(n_neighbors=2),
     4: KNeighborsRegressor(n_neighbors=4),
     15: KNeighborsRegressor(n_neighbors=15)}

Now let's plot all the k values in same plot.

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))

ax.plot(dfcars.wt, dfcars.mpg, 'o', label="data")

xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)

# let's unpack the dictionary to its elements (items) which is the k and Regressor
for k, regressor in regdict.items():
    predictions = regressor.predict(xgrid.reshape(-1,1))
    ax.plot(xgrid, predictions, label="{}-NN".format(k))

ax.legend();
```

Exercise

Explain what you see in the graph. **Hint** Notice how the $1$-NN goes through every point on the training set but utterly fails elsewhere.

Lets look at the scores on the training set.

```python
ks = range(1, 15) # Grid of k's
scores_train = [] # R2 scores
for k in ks:
    # Create KNN model
    knnreg = KNeighborsRegressor(n_neighbors=k)

    # Fit the model to training data
    knnreg.fit(X_train, y_train)

    # Calculate R^2 score
    score_train = knnreg.score(X_train, y_train)
    scores_train.append(score_train)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_train,'o-')
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')
```

Exercise

* Why do we get a perfect $R^2$ at k=1 for the training set?
* Make the same plot as above on the *test* set.
* What is the best $k$?

```python
# Your code here
ks = range(1, 15)
scores_test = []
for k in ks:
    kr = KNeighborsRegressor(n_neighbors=k)
    kr.fit(X_train, y_train)
    score = kr.score(X_test, y_test)
    scores_test.append(score)
fig, ax = plt.subplots(1, 1, figsize=(12, 6))
ax.plot(ks, scores_test, 'o-')
ax.set_xlabel('k')
ax.set_ylabel('R^2 score')
ax.grid('True')
```

```python
# %load solutions/knn_regression.py
ks = range(1, 7) # Grid of k's
scores_test = [] # R2 scores
for k in ks:
    knnreg = KNeighborsRegressor(n_neighbors=k) # Create KNN model
    knnreg.fit(X_train, y_train) # Fit the model to training data
    score_test = knnreg.score(X_test, y_test) # Calculate R^2 score
    scores_test.append(score_test)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_test,'o-', ms=12)
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')

```

```python
# solution to previous exercise
r2_test = knnreg.score(X_test, y_test)
print(f'kNN model with {k} neighbors gives R^2 on the test set: {r2_test:.5}')
```

    kNN model with 6 neighbors gives R^2 on the test set: 0.63189

```python

```


```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import scipy as sp
import scipy.linalg
import sympy as sy
sy.init_printing()
# import warnings
# warnings.filterwarnings("ignore")
```

```python
import linAlgVis
```

#  Basis

**Basis**, denoted as $B$, is the minimum unit of user-customization of coordinates, which is any type of coordinate system other than Cartesian.

##  Basis of $\mathbb{R}^2$

Formally speaking, the basis is a set of vectors $B$ in vector space $V$ with two conditions:
1. All vectors in $B$ are independent.
2. $\text{span}(B)=V$

We have seen _standard basis_ in earlier chapters, for instance in $\mathbb{R}^2$

$$e_1=
\left[
\begin{matrix}
1\\0
\end{matrix}
\right],
\ e_2=\left[
\begin{matrix}
0\\1
\end{matrix}
\right]
$$

and in $\mathbb{R}^3$

$$e_1=
\left[
\begin{matrix}
1\\0\\0
\end{matrix}
\right],
\ e_2=\left[
\begin{matrix}
0\\1\\0
\end{matrix}
\right],
\ e_3=\left[
\begin{matrix}
0\\0\\1
\end{matrix}
\right]
$$

Here we show the linear combination of standard basis for vector $(3, 6)$

```python
fig, ax = plt.subplots(figsize = (10, 10))

arrows = np.array([[[0,0,1,0]],
                  [[0,0,0,1]],
                  [[0,0,3,0]],
                  [[0,0,0,6]],
                  [[0,0,3,6]]])

colors = ['r','b','r','b','g']
for i in range(arrows.shape[0]):
    X,Y,U,V = zip(*arrows[i,:,:])
    ax.arrow(X[0], Y[0], U[0], V[0], color = colors[i], width = .03,
             length_includes_head = True,
             head_width = .2, # default: 3*width
             head_length = .3,
             overhang = .4)

############################Dashed##################################
line1 = np.array([[3, 0], [3, 6]])
ax.plot(line1[:,0], line1[:,1], ls = '--', lw = 3, color = 'black', alpha = .5)

line2 = np.array([[0, 6], [3, 6]])
ax.plot(line2[:,0], line2[:,1], ls = '--', lw = 3, color = 'black', alpha = .5)

############################Text#####################################

ax.text(0, 1, '$e_2$', size = 15)
ax.text(1, 0, '$e_1$', size = 15)

ax.text(0, 6, '$6e_2$', size = 15)
ax.text(3, 0, '$3e_1$', size = 15)

ax.text(3, 6, '$3e_1+6e_2$', size = 15)

###########################Grid Setting##############################
# Major ticks every 20, minor ticks every 5
major_ticks = np.arange(0, 10, 2)
minor_ticks = np.arange(0, 10, .5)

ax.set_xticks(major_ticks)
ax.set_xticks(minor_ticks, minor=True)

ax.set_yticks(major_ticks)
ax.set_yticks(minor_ticks, minor=True)

ax.grid(which='both')
ax.grid(which='minor', alpha=0.2)
ax.grid(which='major', alpha=0.5)

#######################################################################
ax.set_xlabel('x-axis', size = 18)
ax.set_ylabel('y-axis', size = 18)
ax.axis([-1, 10, -1, 10])
ax.grid()
plt.show()
```

But non-standard basis is what we mostly use, we can show that $(2, 1)$ and $(-1, 2)$ is a basis for $\mathbb{R}^2$.

```python
fig, ax = plt.subplots(figsize = (10, 10))

v1 = np.array([2, 1])
v2 = np.array([-1, 2])
v1m2 = 2*v1
v2m3 = 3*v2

arrows = np.array([[[0, 0, v1[0], v1[1]]],
               [[0, 0, v2[0], v2[1]]],
               [[0, 0, 2*v1[0], 2*v1[1]]],
               [[0, 0, 3*v2[0], 3*v2[1]]],
               [[0, 0, (v1m2+v2m3)[0],(v1m2+v2m3)[1]]]])

colors = ['r','b','r','b','g']
for i in range(arrows.shape[0]):
    X,Y,U,V = zip(*arrows[i,:,:])
    ax.arrow(X[0], Y[0], U[0], V[0], color = colors[i], width = .03,
             length_includes_head = True,
             head_width = .2, # default: 3*width
             head_length = .3,
             overhang = .4)

# ############################ Dashed ##################################

point1 = [v2m3[0], v2m3[1]]
point2 = [v2m3[0]+v1m2[0], v2m3[1]+v1m2[1]]
line = np.array([point1, point2])
ax.plot(line[:,0], line[:,1], ls = '--', lw = 3, color = 'black', alpha = .5)

point1 = [v1m2[0], v1m2[1]]
point2 = [v2m3[0]+v1m2[0], v2m3[1]+v1m2[1]]
line = np.array([point1, point2])
ax.plot(line[:,0], line[:,1], ls = '--', lw = 3, color = 'black', alpha = .5)

############################Text#####################################

ax.text(2, 1, '$v_1$', size = 15)
ax.text(-1, 2, '$v_2$', size = 15)

ax.text(v1m2[0],v1m2[1], '$2v_1$', size = 15)
ax.text(v2m3[0],v2m3[1], '$3v_2$', size = 15)

ax.text(v1m2[0]+v2m3[0],v1m2[1]+v2m3[1], '$2v_1+3v_2$', size = 15)
############################## Grid ###############################

t = np.linspace(-6, 6)
for k in range(-6, 7):
    x = 2*k - t
    y = k + 2*t
    ax.plot(x, y, ls = '--', color = 'red', alpha = .3)

for k in range(-6, 7):
    x = -k+2*t
    y = 2*k+t
    ax.plot(x, y, ls = '--', color = 'red', alpha = .3)

#######################################################################
ax.set_xlabel('x-axis', size = 18)
ax.set_ylabel('y-axis', size = 18)
ax.axis([-6,6, 0, 10]) # np.linalg.norm(v1m2+v2m3) is intercept
plt.show()
```

Whether basis is standard or not, as long as they are independent, they span $\mathbb{R}^2$.

##  Basis of $\mathbb{R}^3$

Next we show the standard basis and a non-standard basis of $\mathbb{R}^3$.

The standard basis in $\mathbb{R}^3$ is

$$
(e_1, e_2, e_3)=
\left[
\begin{matrix}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{matrix}
\right]
$$

and we can show a vector $(2,3,4)$ in $\mathbb{R}^3$ is a linear combination of them. We did a 3D linear combination plot in lecture 6, here we just reproduce it by importing the module at the top of the note.

```python
linAlgVis.linearCombo(2,3,4)
```

Next we show the linear combination of a non-standard basis, $(2,1,0), (0,3,1), (0,0,3)$. I also wrote another function ```linearComboNonStd``` in the ```linAlgVis``` module.

```python
a, b, c = 2, 3, 4
vec1 = np.array([2,1,0])
vec2 = np.array([0,3,1])
vec3 = np.array([1,2,3])
linAlgVis.linearComboNonStd(2,3,4, vec1, vec2, vec3)
```

#  Dimension

If $B$ is the basis for $V$, then the **number of vectors** in $B$ is the **dimension** of $V$, denoted as $\text{dim}(V)$.

 Theorem 1

Let $B$ be the basis of $V$, $B$ has $n$ vectors, and $T$ is a set of vectors in $V$, if $T$ has $p$ vectors that $p>n$, then $T$ must be linearly dependent.

 Theorem 2

If $B$ and $T$ both are bases of $V$ then $B$ and $T$ must have the same number of vectors which is the $\text{dim}(V)$.

 Theorem 3

$\text{dim}(V) = n$ and $S$ is a set of vectors from $V$ with $n$ linearly independent vectors, then $\text{span}(S)=V$.

 Theorem 4

Let $v_1, v_2, ...,v_n$ be a set of vectors in the vector space $V$ and let $W = \text{span}\{v_1,v_2,...,v_n\}$. If $v_n$ is a linear combination of $v_1, v_2,...v_{n-1}$, then $W = \text{span}\{v_1,v_2,...,v_{n-1}\}$

These theorems are self-explanatory, no need to memorize, the best way to understand them is visualize them in your mind with $\mathbb{R}^3$.

#  Column Space

Columns space of a matrix is denoted as $\text{Col}A$, which is the space spanned by all columns of a matrix.

 Important Fact

Row operations will not change the dependence of the columns of a matrix.

Let's say we have a matrix $A$

```python
A = sy.Matrix([[1, -2, -1, 3, 0], [-2, 4, 5, -5, 3], [3, -6, -6, 8, 2]]);A
```

$\displaystyle \left[\begin{matrix}1 & -2 & -1 & 3 & 0\\-2 & 4 & 5 & -5 & 3\\3 & -6 & -6 & 8 & 2\end{matrix}\right]$

Perform ```rref``` operations, and dependence of $\text{Col}A$ reserved.

```python
A.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & -2 & 0 & \frac{10}{3} & 0\\0 & 0 & 1 & \frac{1}{3} & 0\\0 & 0 & 0 & 0 & 1\end{matrix}\right], \  \left( 0, \  2, \  4\right)\right)$

The $2nd$ and the $4th$ column are the linear combination of other vectors, it is safe remove them without tampering the column space. Therefore the $\text{Col}A$ is

```python
ColA = sy.Matrix([[1, -1,  0], [-2,  5,  3], [3, -6, 2]]); ColA
```

$\displaystyle \left[\begin{matrix}1 & -1 & 0\\-2 & 5 & 3\\3 & -6 & 2\end{matrix}\right]$

##  Column Spaces Aren't the Same

Did you notice there was a catch when we say the dependency of $\text{Col}A$ was not affect by row operations, however we did not say the column spaces are the same as before and after the row operations.

Actually, they can never be the same.

Consider the matrix $A$:

```python
A = sy.Matrix([[3, -1, -1],[2, 4, 4], [-1, 1, 1]]); A
```

$\displaystyle \left[\begin{matrix}3 & -1 & -1\\2 & 4 & 4\\-1 & 1 & 1\end{matrix}\right]$

If we perform ```rref```, $A$ is turned into $B$. Apparently the column space of them are different apparently.

```python
B = A.rref(); B
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 1\\0 & 0 & 0\end{matrix}\right], \  \left( 0, \  1\right)\right)$

Now list the basis of their column spaces: are $\text{Col}A$ and $\text{Col}B$ the same?

$$
\text{col}(A)=\text{span}\left\{\left[ \matrix{3\cr 2\cr -1}\right],\ \left[\matrix{-1\cr 4\cr 1}\right]\right\}
$$
$$
\text{col}(B)=\text{span}\left\{\left[ \matrix{1\cr 0\cr 0}\right],\ \left[\matrix{0\cr 1\cr 0}\right]\right\}
$$

It's easy to visualize them, they are two intersecting planes.

```python
fig = plt.figure(figsize = (10,10))
ax = fig.add_subplot(projection='3d')

s = np.linspace(-2, 2, 20)
t = np.linspace(-2, 2, 20)
S, T = np.meshgrid(s, t)

X = 3*S - T
Y = 2*S + 4*T
Z = -S + T
ax.plot_wireframe(X, Y, Z, linewidth = .5, color = 'r')

s = np.linspace(-10, 10, 20)
t = np.linspace(-10, 10, 20)
S, T = np.meshgrid(s, t)
X = S
Y = T
Z = np.zeros(S.shape)
ax.plot_wireframe(X, Y, Z, linewidth = .5, color = 'k')
ax.view_init(elev=14, azim=58)
```

##  Method for Finding Basis of $\mathbb{R}^n$

Consider matrix $A_{4\times 2}$, find a basis for $\mathbb{R}^4$.

Note that we only have two column vectors, not possible to span $\mathbb{R}^4$. The common method is to use another two standard basis vectors joined with $A$ to form the basis of $\mathbb{R}^4$.

```python
A = sy.randMatrix(4, 2);A
```

$\displaystyle \left[\begin{matrix}45 & 33\\79 & 19\\45 & 75\\3 & 27\end{matrix}\right]$

```python
I = sy.eye(4);I
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

Concatenate the matrix horizontally.

```python
Aug = A.row_join(I);Aug# concatenate
```

$\displaystyle \left[\begin{matrix}45 & 33 & 1 & 0 & 0 & 0\\79 & 19 & 0 & 1 & 0 & 0\\45 & 75 & 0 & 0 & 1 & 0\\3 & 27 & 0 & 0 & 0 & 1\end{matrix}\right]$

```python
Aug.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0 & 0 & \frac{3}{110} & - \frac{5}{66}\\0 & 1 & 0 & 0 & - \frac{1}{330} & \frac{1}{22}\\0 & 0 & 1 & 0 & - \frac{62}{55} & \frac{21}{11}\\0 & 0 & 0 & 1 & - \frac{346}{165} & \frac{169}{33}\end{matrix}\right], \  \left( 0, \  1, \  2, \  3\right)\right)$

Therefore the first 4 columns of augmented matrix is a basis.

```python
Aug[:,0:4]
```

$\displaystyle \left[\begin{matrix}45 & 33 & 1 & 0\\79 & 19 & 0 & 1\\45 & 75 & 0 & 0\\3 & 27 & 0 & 0\end{matrix}\right]$

#  Change of Basis

Change of basis is for changing the coordinate system,e.g. from Cartesian to other system.

##  Coordinate Vector

We have a basis $B=\{b_1, b_2\}$, and $B$'s **coordinate vector** is denoted as $\big[v\big]_B$. If

$$
b_1 = \left[\begin{matrix}1\\1 \end{matrix}\right]\\
b_2 = \left[\begin{matrix}-2\\1 \end{matrix}\right]\\
\big[v\big]_B = \left[\begin{matrix}2\\3 \end{matrix}\right]
$$

Then compute $v$ by using the basis and its coordinate vectors. We want to convert the $\big[v\big]_B$ into a pair of Cartesian coordinates.

```python
P = sy.Matrix([[1,-2],[1,1]]); P
```

$\displaystyle \left[\begin{matrix}1 & -2\\1 & 1\end{matrix}\right]$

```python
v_B = sy.Matrix([[2], [3]]); v_B
```

$\displaystyle \left[\begin{matrix}2\\3\end{matrix}\right]$

```python
P*v_B
```

$\displaystyle \left[\begin{matrix}-4\\5\end{matrix}\right]$

Now we have successfully converted  $\big[v\big]_B$  into Cartesian

On the other way around, what if we know $v$ i.e. the Cartesian coordinates, and we want to find out $\big[v\big]_B$, for instance

$$
v = \left[\begin{matrix}-1\\-4 \end{matrix}\right]
$$

Then we can formulate the question as:

$$
\left[\begin{matrix}1 & -2\\1 & 1\end{matrix}\right]
\left[\begin{matrix}b_1\\b_2 \end{matrix}\right]=
\left[\begin{matrix}-1\\-4 \end{matrix}\right]
$$

Just solve the linear system

```python
Aug = sy.Matrix([[1,-2,-1],[1,1,-4]])
Aug.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & -3\\0 & 1 & -1\end{matrix}\right], \  \left( 0, \  1\right)\right)$

We get $\big[v\big]_B = (-3, -1)^T$.

##  Change of Basis

Most probably we would like to change coordinates from  $\big[v\big]_B$ to  $\big[v\big]_C$, which both aren't Cartesian coordinates.

Say we have two bases $B=\{b_1,\ b_2\}$, $C=\{c_1,\ c_2\}$ in $\mathbb{R}^2$.

$$
b_1=
\left[\begin{matrix}2\\1 \end{matrix}\right],\quad
b_2 =
\left[\begin{matrix}-1\\1 \end{matrix}\right]\\
c_1=
\left[\begin{matrix}1\\1 \end{matrix}\right],\quad
c_2 =
\left[\begin{matrix}-3\\1 \end{matrix}\right]\\
$$

We would like to change $\big[v\big]_B$ to $\big[v\big]_C$. Formally this is reformulated as finding **transition matrix** $P_{C\leftarrow B}$.

First we find coordinate vectors for $b_1$ and $b_2$ in $C: b_1 = xc_1+yc_2$, $b_2 = xc_1+yc_2$.

$$
\left[\begin{matrix}2\\1 \end{matrix}\right] = x\left[\begin{matrix}1\\1 \end{matrix}\right]+y\left[\begin{matrix}-3\\1 \end{matrix}\right]= \left[\begin{matrix}1 & -3\\1 &1 \end{matrix}\right]\left[\begin{matrix}x \\y \end{matrix}\right]\\
\left[\begin{matrix}-1\\1 \end{matrix}\right] = x\left[\begin{matrix}1\\1 \end{matrix}\right]+y\left[\begin{matrix}-3\\1 \end{matrix}\right]=\left[\begin{matrix}1 & -3\\1 &1 \end{matrix}\right]\left[\begin{matrix}x \\y \end{matrix}\right]\\
$$

Extract augmented matrix for both system:

$$
\left[\begin{matrix}1 & -3 & 2\\1 &1 &1 \end{matrix}\right]\\
\left[\begin{matrix}1 & -3 & -1\\1 &1 &1 \end{matrix}\right]
$$

They can be combined into one augmented matrix $[C|B]$

$$
\left[\begin{matrix}1 & -3  & 2 & -1\\1 &1 &1 &1 \end{matrix}\right]
$$

```python
Aug = sy.Matrix([[1,-3,2,-1],[1, 1, 1, 1]])
Aug.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & \frac{5}{4} & \frac{1}{2}\\0 & 1 & - \frac{1}{4} & \frac{1}{2}\end{matrix}\right], \  \left( 0, \  1\right)\right)$

The last two columns of rref augmented matrix is the $P_{C\leftarrow B}$.

In general, we can find transition matrix by using

$$[\ C\ |\ B\ ]\longrightarrow [\ I\ |\ P_{C\leftarrow B}\ ]$$

###  An Example

Let

$$
B=\left\{\left[\begin{array}{l}
1 \\
1 \\
0
\end{array}\right],\left[\begin{array}{l}
0 \\
1 \\
1
\end{array}\right],\left[\begin{array}{l}
0 \\
1 \\
0
\end{array}\right]\right\} \quad \text { and } \quad C=\left\{\left[\begin{array}{l}
1 \\
1 \\
2
\end{array}\right],\left[\begin{array}{l}
2 \\
1 \\
1
\end{array}\right],\left[\begin{array}{l}
1 \\
0 \\
1
\end{array}\right]\right\}
$$

be two bases for $\mathbb{R}^3$, find the $P_{C\leftarrow B}$ and given $\big[v\big]_B = (1,-1,2)$, find $\big[v\big]_C$.

Construct augmented matrix:

```python
v_B =  sy.Matrix([1, -1, 2])
B = sy.Matrix([[1, 0, 0], [1, 1 ,1], [0, 1, 0]])
C = sy.Matrix([[1, 2, 1], [1, 1 ,0], [2, 1, 1]])
Aug = C.row_join(B); Aug# concatenate
```

$\displaystyle \left[\begin{matrix}1 & 2 & 1 & 1 & 0 & 0\\1 & 1 & 0 & 1 & 1 & 1\\2 & 1 & 1 & 0 & 1 & 0\end{matrix}\right]$

The RHS block of rref is $P_{C\leftarrow B}$ as we have shown above.

```python
Aug_rref = Aug.rref();Aug_rref
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0 & 0 & 1 & \frac{1}{2}\\0 & 1 & 0 & 1 & 0 & \frac{1}{2}\\0 & 0 & 1 & -1 & -1 & - \frac{3}{2}\end{matrix}\right], \  \left( 0, \  1, \  2\right)\right)$

```python
v_C = Aug_rref[0][:,3:6]*v_B; v_C
```

$\displaystyle \left[\begin{matrix}0\\2\\-3\end{matrix}\right]$

The $\big[v\big]_C$ is $(0, 2, -3)$.


# Converting *Exact* ADM Initial Data in the Spherical or Cartesian Basis to BSSN Initial Data in the Desired Curvilinear Basis
## Author: Zach Etienne
### Formatting improvements courtesy Brandon Clark

[comment]: <> (Abstract: TODO)

### This module is meant for use only with initial data that can be represented exactly in ADM form, either in the Spherical or Cartesian basis. I.e., the ADM variables are given $\left\{\gamma_{ij}, K_{ij}, \alpha, \beta^i\right\}$ *exactly* as functions of $(r,\theta,\phi)$ or $(x,y,z)$, respectively. If instead the initial data are given only numerically (e.g., through an initial data solver), then [the Numerical-ADM-Spherical/Cartesian-to-BSSNCurvilinear module](Tutorial-ADM_Initial_Data-Converting_Numerical_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.ipynb) will need to be used instead.

**Module Status:**  Self-Validated

**Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). **Additional validation tests may have been performed, but are as yet, undocumented. (TODO)**

### NRPy+ Source Code for this module: [BSSN/ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear.py](../edit/BSSN/ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear.py)

## Introduction:
Given the ADM variables:

$$\left\{\gamma_{ij}, K_{ij}, \alpha, \beta^i\right\}$$

in the Spherical or Cartesian basis, and as functions of $(r,\theta,\phi)$ or $(x,y,z)$, respectively, this module documents their conversion to the BSSN variables

$$\left\{\bar{\gamma}_{i j},\bar{A}_{i j},\phi, K, \bar{\Lambda}^{i}, \alpha, \beta^i, B^i\right\},$$

in the desired curvilinear basis (given by `reference_metric::CoordSystem`). Then it rescales the resulting BSSNCurvilinear variables (as defined in [the covariant BSSN formulation tutorial](Tutorial-BSSN_formulation.ipynb)) into the form needed for solving Einstein's equations with the BSSN formulation:

$$\left\{h_{i j},a_{i j},\phi, K, \lambda^{i}, \alpha, \mathcal{V}^i, \mathcal{B}^i\right\}.$$

We will use as our core example in this module UIUC initial data, which are ([as documented in their NRPy+ initial data module](Tutorial-ADM_Initial_Data-UIUC_BlackHole.ipynb)) given in terms of ADM variables in Spherical coordinates.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Initialize core Python/NRPy+ modules
1. [Step 2](#cylindrical): Desired output BSSN Curvilinear coordinate system set to Cylindrical, as a proof-of-principle
1. [Step 3](#admxx0xx1xx2): Converting ADM variables to functions of (`xx0,xx1,xx2`)
1. [Step 4](#adm_jacobian): Applying Jacobian transformations to get in the correct `xx0,xx1,xx2` basis
1. [Step 5](#adm2bssn): Call functions within [`BSSN.BSSN_in_terms_of_ADM`](../edit/BSSN/BSSN_in_terms_of_ADM.py) ([**tutorial**](Tutorial-BSSN_in_terms_of_ADM.ipynb)) to perform the ADM-to-BSSN conversion
1. [Step 6](#code_validation): Code Validation against `BSSN.ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear` NRPy+ module
1. [Step 7](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Initialize core Python/NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

```python
# Step P1: Import needed NRPy+ core modules:
import sympy as sp                # SymPy: The Python computer algebra package upon which NRPy+ depends
from outputC import *             # NRPy+: Core C code output module
import indexedexp as ixp          # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm    # NRPy+: Reference metric support
import NRPy_param_funcs as par    # NRPy+: parameter interface
import BSSN.BSSN_quantities as Bq # NRPy+: The EvolvedConformalFactor_cf parameter is used below
```

# Step 2: Desired output BSSN Curvilinear coordinate system set to Cylindrical, as a proof-of-principle \[Back to [top](#toc)\]
$$\label{cylindrical}$$

```python
# The ADM & BSSN formalisms only work in 3D; they are 3+1 decompositions of Einstein's equations.
#    To implement axisymmetry or spherical symmetry, simply set all spatial derivatives in
#    the relevant angular directions to zero; DO NOT SET DIM TO ANYTHING BUT 3.

# Step P1: Set spatial dimension (must be 3 for BSSN)
DIM = 3

# Set the desired *output* coordinate system to Cylindrical:
par.set_parval_from_str("reference_metric::CoordSystem","Cylindrical")
rfm.reference_metric()

# Import UIUC Black Hole initial data
import BSSN.UIUCBlackHole as uibh
uibh.UIUCBlackHole(ComputeADMGlobalsOnly=True)
Sph_r_th_ph_or_Cart_xyz = [uibh.r,uibh.th,uibh.ph]
alphaSphorCart    = uibh.alphaSph
betaSphorCartU    = uibh.betaSphU
BSphorCartU       = uibh.BSphU
gammaSphorCartDD  = uibh.gammaSphDD
KSphorCartDD      = uibh.KSphDD
```

# Step 3: Converting ADM variables to functions of ${\rm xx0},{\rm xx1},{\rm xx2}$ \[Back to [top](#toc)\]
$$\label{admxx0xx1xx2}$$

ADM variables are given as functions of $(r,\theta,\phi)$ or $(x,y,z)$. We convert them to functions of `(xx0,xx1,xx2)` using SymPy's `subs()` function.

```python
# Step 3: All input quantities are in terms of r,th,ph or x,y,z. We want them in terms
#         of xx0,xx1,xx2, so here we call sympify_integers__replace_rthph() to replace
#         r,th,ph or x,y,z, respectively, with the appropriate functions of xx0,xx1,xx2
#         as defined for this particular reference metric in reference_metric.py's
#         xxSph[] or xxCart[], respectively:

# UIUC Black Hole initial data are given in Spherical coordinates.
CoordType_in = "Spherical"

# Make sure that rfm.reference_metric() has been called.
#    We'll need the variables it defines throughout this module.
if rfm.have_already_called_reference_metric_function == False:
    print("Error. Called Convert_Spherical_ADM_to_BSSN_curvilinear() without")
    print("       first setting up reference metric, by calling rfm.reference_metric().")
    exit(1)

# Note that substitution only works when the variable is not an integer. Hence the
#         if isinstance(...,...) stuff:
def sympify_integers__replace_rthph_or_Cartxyz(obj, rthph_or_xyz, rthph_or_xyz_of_xx):
    if isinstance(obj, int):
        return sp.sympify(obj)
    else:
        return obj.subs(rthph_or_xyz[0], rthph_or_xyz_of_xx[0]).\
            subs(rthph_or_xyz[1], rthph_or_xyz_of_xx[1]).\
            subs(rthph_or_xyz[2], rthph_or_xyz_of_xx[2])

r_th_ph_or_Cart_xyz_of_xx = []
if CoordType_in == "Spherical":
    r_th_ph_or_Cart_xyz_of_xx = rfm.xxSph
elif CoordType_in == "Cartesian":
    r_th_ph_or_Cart_xyz_of_xx = rfm.xxCart
else:
    print("Error: Can only convert ADM Cartesian or Spherical initial data to BSSN Curvilinear coords.")
    exit(1)

alphaSphorCart = sympify_integers__replace_rthph_or_Cartxyz(
    alphaSphorCart, Sph_r_th_ph_or_Cart_xyz, r_th_ph_or_Cart_xyz_of_xx)
for i in range(DIM):
    betaSphorCartU[i] = sympify_integers__replace_rthph_or_Cartxyz(
        betaSphorCartU[i], Sph_r_th_ph_or_Cart_xyz, r_th_ph_or_Cart_xyz_of_xx)
    BSphorCartU[i]    = sympify_integers__replace_rthph_or_Cartxyz(
        BSphorCartU[i],    Sph_r_th_ph_or_Cart_xyz, r_th_ph_or_Cart_xyz_of_xx)
    for j in range(DIM):
        gammaSphorCartDD[i][j] = sympify_integers__replace_rthph_or_Cartxyz(
            gammaSphorCartDD[i][j], Sph_r_th_ph_or_Cart_xyz, r_th_ph_or_Cart_xyz_of_xx)
        KSphorCartDD[i][j]     = sympify_integers__replace_rthph_or_Cartxyz(
            KSphorCartDD[i][j],     Sph_r_th_ph_or_Cart_xyz, r_th_ph_or_Cart_xyz_of_xx)
```

# Step 4:  Applying Jacobian transformations to get in the correct `xx0,xx1,xx2` basis \[Back to [top](#toc)\]
$$\label{adm_jacobian}$$

All ADM initial data quantities are now functions of `xx0,xx1,xx2`, but they are still in the Spherical or Cartesian basis. We can now directly apply Jacobian transformations to get them in the correct `xx0,xx1,xx2` basis. The following discussion holds for either Spherical or Cartesian input data, so for simplicity let's just assume the data are given in Spherical coordinates.

All ADM tensors and vectors are in the Spherical coordinate basis $x^i_{\rm Sph} = (r,\theta,\phi)$, but we need them in the curvilinear coordinate basis $x^i_{\rm rfm}=$`(xx0,xx1,xx2)` set by the `"reference_metric::CoordSystem"` variable. Empirically speaking, it is far easier to write `(x(xx0,xx1,xx2),y(xx0,xx1, xx2),z(xx0,xx1,xx2))` than the inverse, so we will compute the Jacobian matrix

$$
{\rm Jac\_dUSph\_dDrfmUD[i][j]} = \frac{\partial x^i_{\rm Sph}}{\partial x^j_{\rm rfm}},
$$

via exact differentiation (courtesy SymPy), and the inverse Jacobian
$$
{\rm Jac\_dUrfm\_dDSphUD[i][j]} = \frac{\partial x^i_{\rm rfm}}{\partial x^j_{\rm Sph}},
$$

using NRPy+'s `generic_matrix_inverter3x3()` function. In terms of these, the transformation of BSSN tensors from Spherical to `"reference_metric::CoordSystem"` coordinates may be written:

\begin{align}
\beta^i_{\rm rfm} &= \frac{\partial x^i_{\rm rfm}}{\partial x^\ell_{\rm Sph}} \beta^\ell_{\rm Sph}\\
B^i_{\rm rfm} &= \frac{\partial x^i_{\rm rfm}}{\partial x^\ell_{\rm Sph}} B^\ell_{\rm Sph}\\
\gamma^{\rm rfm}_{ij} &=
\frac{\partial x^\ell_{\rm Sph}}{\partial x^i_{\rm rfm}}
\frac{\partial x^m_{\rm Sph}}{\partial x^j_{\rm rfm}} \gamma^{\rm Sph}_{\ell m}\\
K^{\rm rfm}_{ij} &=
\frac{\partial x^\ell_{\rm Sph}}{\partial x^i_{\rm rfm}}
\frac{\partial x^m_{\rm Sph}}{\partial x^j_{\rm rfm}} K^{\rm Sph}_{\ell m}
\end{align}

```python
# Step 2: All ADM initial data quantities are now functions of xx0,xx1,xx2, but
#         they are still in the Spherical or Cartesian basis. We can now directly apply
#         Jacobian transformations to get them in the correct xx0,xx1,xx2 basis:

# alpha is a scalar, so no Jacobian transformation is necessary.
alpha = alphaSphorCart

Jac_dUSphorCart_dDrfmUD = ixp.zerorank2()
for i in range(DIM):
    for j in range(DIM):
        Jac_dUSphorCart_dDrfmUD[i][j] = sp.diff(r_th_ph_or_Cart_xyz_of_xx[i],rfm.xx[j])

Jac_dUrfm_dDSphorCartUD, dummyDET = ixp.generic_matrix_inverter3x3(Jac_dUSphorCart_dDrfmUD)

betaU   = ixp.zerorank1()
BU      = ixp.zerorank1()
gammaDD = ixp.zerorank2()
KDD     = ixp.zerorank2()
for i in range(DIM):
    for j in range(DIM):
        betaU[i] += Jac_dUrfm_dDSphorCartUD[i][j] * betaSphorCartU[j]
        BU[i]    += Jac_dUrfm_dDSphorCartUD[i][j] * BSphorCartU[j]
        for k in range(DIM):
            for l in range(DIM):
                gammaDD[i][j] += Jac_dUSphorCart_dDrfmUD[k][i]*Jac_dUSphorCart_dDrfmUD[l][j] * gammaSphorCartDD[k][l]
                KDD[i][j]     += Jac_dUSphorCart_dDrfmUD[k][i]*Jac_dUSphorCart_dDrfmUD[l][j] *     KSphorCartDD[k][l]
```

# Step 5: Call functions within [`BSSN.BSSN_in_terms_of_ADM`](../edit/BSSN/BSSN_in_terms_of_ADM.py) ([**tutorial**](Tutorial-BSSN_in_terms_of_ADM.ipynb)) to perform the ADM-to-BSSN conversion \[Back to [top](#toc)\]
$$\label{adm2bssn}$$

All ADM quantities were input into this function in the Spherical or Cartesian basis, as functions of $r,\theta,\phi$ or $x,y,z$, respectively. In [Step 3](#admxx0xx1xx2) and [Step 4](#adm_jacobian) above, we converted them to the `xx0,xx1,xx2` basis, and as functions of `xx0,xx1,xx2`. Here we convert ADM 3-metric, extrinsic curvature, and gauge quantities in the `xx0,xx1,xx2` (a.k.a. "rfm") basis to their BSSN Curvilinear counterparts, in the same basis.

```python
import BSSN.BSSN_in_terms_of_ADM as BitoA
BitoA.gammabarDD_hDD(                   gammaDD)
BitoA.trK_AbarDD_aDD(                   gammaDD,KDD)
BitoA.LambdabarU_lambdaU__exact_gammaDD(gammaDD)
BitoA.cf_from_gammaDD(                  gammaDD)
BitoA.betU_vetU(                        betaU,BU)
hDD     = BitoA.hDD
trK     = BitoA.trK
aDD     = BitoA.aDD
lambdaU = BitoA.lambdaU
cf      = BitoA.cf
vetU    = BitoA.vetU
betU    = BitoA.betU
```

# Step 6: Code Validation against `BSSN.ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear` module \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for BrillLindquist initial data between
1. this tutorial and
2. the NRPy+ [BSSN.ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear](../edit/BSSN/ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear.py) module.

By default, we analyze these expressions in Spherical coordinates, though other coordinate systems may be chosen.

```python
import BSSN.UIUCBlackHole as uibh
import BSSN.ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear as ADMtoBSSN
returnfunction = uibh.UIUCBlackHole()
mod_cf,mod_hDD,mod_lambdaU,mod_aDD,mod_trK,mod_alpha,mod_vetU,mod_betU = \
    ADMtoBSSN.Convert_Spherical_or_Cartesian_ADM_to_BSSN_curvilinear("Spherical",uibh.Sph_r_th_ph,
                                            uibh.gammaSphDD, uibh.KSphDD, uibh.alphaSph, uibh.betaSphU, uibh.BSphU)

print("Consistency check between this tutorial notebook and BSSN.ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear NRPy+ module: ALL SHOULD BE ZERO.")

print("cf - mod_cf = " + str(cf - mod_cf))
print("trK - mod_trK = " + str(trK - mod_trK))
print("alpha - mod_alpha = " + str(alpha - mod_alpha))

for i in range(DIM):
    print("vetU["+str(i)+"] - mod_vetU["+str(i)+"] = " + str(vetU[i] - mod_vetU[i]))
    print("betU["+str(i)+"] - mod_betU["+str(i)+"] = " + str(betU[i] - mod_betU[i]))
    print("lambdaU["+str(i)+"] - mod_lambdaU["+str(i)+"] = " + str(lambdaU[i] - mod_lambdaU[i]))
    for j in range(DIM):
        print("hDD["+str(i)+"]["+str(j)+"] - mod_hDD["+str(i)+"]["+str(j)+"] = "
              + str(hDD[i][j] - mod_hDD[i][j]))
        print("aDD["+str(i)+"]["+str(j)+"] - mod_aDD["+str(i)+"]["+str(j)+"] = "
              + str(aDD[i][j] - mod_aDD[i][j]))

# If you wish to generate & analyze C code output, uncomment the following:
# import os, shutil            # Standard Python modules for multiplatform OS-level functions
# import cmdline_helper as cmd # NRPy+: Multi-platform Python command-line interface
# # Step P2: Create C code output directory:
# Ccodesdir = os.path.join("BSSN_Exact_ADM_validation/")
# # First remove C code output directory if it exists
# # Courtesy https://stackoverflow.com/questions/303200/how-do-i-remove-delete-a-folder-that-is-not-empty
# # !rm -r ScalarWaveCurvilinear_Playground_Ccodes
# shutil.rmtree(Ccodesdir, ignore_errors=True)
# # Then create a fresh directory
# cmd.mkdir(Ccodesdir)
# with open(os.path.join(Ccodedir,"UIUCBlackHole-CylindricalTest.h"),"w") as file:
#     file.write(uibh.returnfunction)
```

    Consistency check between this tutorial notebook and BSSN.ADM_Exact_Spherical_or_Cartesian_to_BSSNCurvilinear NRPy+ module: ALL SHOULD BE ZERO.
    cf - mod_cf = 0
    trK - mod_trK = 0
    alpha - mod_alpha = 0
    vetU[0] - mod_vetU[0] = 0
    betU[0] - mod_betU[0] = 0
    lambdaU[0] - mod_lambdaU[0] = 0
    hDD[0][0] - mod_hDD[0][0] = 0
    aDD[0][0] - mod_aDD[0][0] = 0
    hDD[0][1] - mod_hDD[0][1] = 0
    aDD[0][1] - mod_aDD[0][1] = 0
    hDD[0][2] - mod_hDD[0][2] = 0
    aDD[0][2] - mod_aDD[0][2] = 0
    vetU[1] - mod_vetU[1] = 0
    betU[1] - mod_betU[1] = 0
    lambdaU[1] - mod_lambdaU[1] = 0
    hDD[1][0] - mod_hDD[1][0] = 0
    aDD[1][0] - mod_aDD[1][0] = 0
    hDD[1][1] - mod_hDD[1][1] = 0
    aDD[1][1] - mod_aDD[1][1] = 0
    hDD[1][2] - mod_hDD[1][2] = 0
    aDD[1][2] - mod_aDD[1][2] = 0
    vetU[2] - mod_vetU[2] = 0
    betU[2] - mod_betU[2] = 0
    lambdaU[2] - mod_lambdaU[2] = 0
    hDD[2][0] - mod_hDD[2][0] = 0
    aDD[2][0] - mod_aDD[2][0] = 0
    hDD[2][1] - mod_hDD[2][1] = 0
    aDD[2][1] - mod_aDD[2][1] = 0
    hDD[2][2] - mod_hDD[2][2] = 0
    aDD[2][2] - mod_aDD[2][2] = 0

# Step 7: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename [Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.pdf](Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.ipynb
!pdflatex -interaction=batchmode Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.tex
!pdflatex -interaction=batchmode Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.tex
!pdflatex -interaction=batchmode Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    [NbConvertApp] Converting notebook Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.ipynb to latex
    [NbConvertApp] Writing 49811 bytes to Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.tex
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode