# Python refresher: Part II

Last week, we have seen some basic python, which you should be familiar with by now. This includes comments, data types and some of the operators that can be used with them, and other tids and bits.

This week, we're going to look at some slightly more complex topics that are needed to work with the code in the second section of the course. Specifically, some concepts that we need to use Piantadosi's LOTlib3 library.

I expect the topics in this lab will be familiar to different extents to different people. Therefore, rather than explaining everything in huge detail, it's easier if you just ask if you get stuck on a point, and we explore it more in detail.

## Importing and using libraries

### What is a library?

Roughly, a library is a collection of useful stuff. This can include functions / variables / classes / other stuff. For instance, to do maths in python we usually import the 'numpy' library, to do plotting we use 'matplotlib' or 'seaborn', etc. LOTlib3, which we will use in the second half of this course, is one such library.

Using libraries allows us to not rewrite stuff that someone else has already written.

### How do we import and use libraries?

To import a library we write `import` follows by the name of the library, as follows:
```python
import numpy
```

To use the stuff in the library we use the library name followed by a dot followed by the name of what we want from the library. For instance:
```python
numpy.pi
```
to get the number $\pi$.

However, often we do not want to use the full name of the library, but rather some abbreviation. In this case, we can write as follows:
```python
import numpy as np
```

```python
import numpy as np
```

and then we can just write:
```python
np.pi
```
which is shorter.

```python
np.pi
```

    3.141592653589793

If we do not want the full library but rather just one thing from it, we can write:
```python
from numpy import pi
```
and then we can just use `pi` directly without needing the `np.`.

```python
from numpy import pi
```

```python
pi
```

    3.141592653589793

## Functions

### Motivation

Often when programming you repeatedly need some code that 'does the same thing', e.g. multiply two numbers, or take a string and transform it in some way. You can think of a function in python as a way of dealing with this without having to repeatedly write the same code (this is not all that a function is good for, but it is one thing that functions are good for). Basically, you give a name to the 'piece of code' that you repeatedly need, and use the name instead of the code itself.

Let's look at an example to understand this better. Suppose you have to write some code that takes a string, changes the first letter to 'A', and returns the first 10 characters. You can write it as follows:

```python
string = 'hello world'
string = 'A' + string[1:]
string = string[:10]
print(string)
```

    Aello worl

Now suppose that you need to perform the procedure above for each element in a list of strings. You can do this by putting the code above in a loop:

```python
strings = ['hello world', 'conversation world', 'goodbye world']
results = []
for string in strings:
    string = 'A' + string[1:]
    string = string[:10]
    results.append(string)
print(results)
```

    ['Aello worl', 'Aonversati', 'Aoodbye wo']

But now suppose that you don't know a priori whether you're getting a list or a single word, and you need to adapt. One natural way to do it would be:

```python
variable = 'bla bla bla'

# check if the variable is a list
if type(variable) is list:
    for string in variable:
        string = 'A' + string[1:]
        string = string[:10]
        results.append(string)
    print(results)
else:
    variable = 'A' + variable[1:]
    variable = variable[:10]
    print(variable)
```

    Ala bla bl

But notes that some lines are repeated, namely
```python
string = 'A' + string[1:]
string = string[:10]
```
We would like a way to only write them once and use them in two places. A natural solution to this is a function that takes a string and performs the transformation we want.

### Standard function definition in python

To define a function in python we do the following:
- start with the keyword `def`
- followed by the name of the function
- followed by an open `(`
- followed by zero or more arguments divided by commas
- followed by `):`
- followed by one or more lines at the next indentation level
- In the points where you want a function to return a value, you can optionally put a `return` statement followed by the value you want the function to return.

For instance:
```python
def name(argument1, argument2):
    first line in function
    second line in function
    return value
```

To solve the problem above we could use:

```python
def transform_string(string_to_transform):
    string_to_transform = 'A' + string_to_transform[1:]
    string_to_transform = string_to_transform[:10]
    return string_to_transform
```

### Calling a function

Then, we can rewrite the above as:

```python
variable = 'bla bla bla'

# check if the variable is a list
if type(variable) is list:
    for string in variable:
        results.append(transform_string(string))
    print(results)
else:
    print(transform_string(string))
```

    Aoodbye wo

Basically what we have done it _called_ a function on a certain input (`transform_string(string)`), which is the same as running the input through the code in the function and getting the return values of the function.

Now we only repeat one word rather than several lines, and also if it turns out we want to change the transformation later, we only need to do it once rather than changing it independently multiple times!

### $\lambda$ ('lambda') functions

At this point it is worth mentioning that there is another notation for defining functions that doesn't use the `def` keyword. Namely, we write `lambda` followed by a space, zero or more comma-separated arguments, a colon, and a single statement.

For example, suppose we want to sum two variables. We could define a function that does it as follows:

```python
sum_function = lambda x, y: x+y

print(sum_function(2, 3))
```

    5

One case where this might be useful is in conjunction with `map`, which is an operation that takes two arguments, a function and a list, and applies the function to each element of the list. E.g., in the following example:

```python
list(map(
    lambda x: x[0]+x[1],
    [[1,2], [4, 4], [1,3]]
))
```

    [3, 8, 4]

Lambda functions are usually a slightly more advanced topic, but they'll come up a lot in the rest of the course, so it's good to start getting used to them already.

### Recursive functions

This is a slighlty mindbending concept: functions can _call themselves_. This means that inside a function $f$, $f$ itself can be called and the return value can be used inside $f$. Of course, this can easily lead to functions that never terminate, but rather keep calling themselves, forever increasing the level of nested calls. For instance, try to think what is happening here:

```python
def bad_recursive_function(x):
    y = x+1
    bad_recursive_function(y)

bad_recursive_function(1)
```

Thankfully, python has a way of preventing the function running infinitely and automatically raises an error after a certain number of nested calls. You can see it here as `RecursionError: maximum recursion depth exceeded`.

Usually, recursive functions are conceptualized as follows:
- There is one or more 'base case', i.e. an input for which the function simply calculates a return value without calling the function itself.
- There are 'complex' cases where the input to the function is somehow 'simplified' and the function is called with the simplified input. Eventually after enough 'simplification' steps the input reduces to a base case, and recursion stops.

A classic case of recursion is the following formulation of the $n$th Fibonacci numbers:
- If $n$ is 0 or 1 (i.e., the first or second Fibonacci number), return 1
- If $n$ is greater than 1, return the $n$th-2 Fibonacci number + the $n$th-1 Fibonacci number

```python
def calculate_fibonacci(n):
    # there are two base cases here!
    if n==0 or n==1:
        # in both base cases, just return 1
        return 1
    else:
        # otherwise, follow the recursive rule
        return calculate_fibonacci(n-2) + calculate_fibonacci(n-1)

for i in range(30):
    print(calculate_fibonacci(i), end=',')
```

    1,1,2,3,5,8,13,21,34,55,89,144,233,377,610,987,1597,2584,4181,6765,10946,17711,28657,46368,75025,121393,196418,317811,514229,832040,

There is a lot to say about recursive functions and how they work, but for now try to really understand how the one above works. If you really understand that simple case, other ones are going to be easy.

## Functions as objects

We have refreshed briefly how to define functions above. Functions in python are _first class citizens_, meaning that they can be treated as objects themselves: passed as arguments of functions, destroyed, transformed in all the same ways as other objects.

Let's see a simple example. Suppose that you have a function $f$ that takes an integer and returns an integer, but you don't know exactly what it does to that integer. But whatever it does, you want another function that does the same thing as $f$ but then adds 3 to the result.

Well, we can define a function $g$ that takes a function $f$ _as an argument_ and returns another function from integer to integer, that first calls $f$ on the integer and then adds three:

```python
def modify_f(func):
    # define a function that runs func and then adds 3
    def g(n):
        return func(n) + 3
    # return that function
    return g
```

Let's see `modify_f` in action:

```python
# define an f that take an int and returns an int
# (namely, the argument multiplied by two)
f = lambda x: x * 2

# get a modified function by running
# g through modify_f
modified_f = modify_f(f)

# call the modified function
# on an example
modified_f(4)
```

    11

The point of this example is that `modify_f` takes _a function_ as an input and return _a function_ as an output. You can use manipulate functions like that!

## Recursion without recursion (note: more difficult!)

(For more explanation on the material in this section, see [this page](https://www.lambdacircle.com/the-y-and-z-combinators-in-python/))

Note that things become complicated easily when we work with functions that take other functions as inputs. In those cases, you can define functions that _take themselves_ (as well as other arguments). This actually allows us to do surprising things, like have the same power as recursion without using recursion directly, i.e., without calling a function in its own definition.

For instance, consider the following function, which as you can see is not strictly speaking recursive:

```python
def strange_fibonacci(f):
    # takes a function f and
    # defines a new function
    # whose behaviour depends on
    # the input function f
    def g(n):
        if n == 0 or n == 1:
            return 1
        else:
            return f(f)(n-1) + f(f)(n-2)
    return g
```

And now consider the following function which in a way encodes the recursion bit (but without being explicitly recursive: it's not calling itself in its own definition):

```python
# takes a function, calls the function on itself
recursive_function = lambda f: f(f)
```

What `recursive_function` does is call a function on itself.

When called with `strange_fibonacci` as an argument, it will effectively calculate `strange_fibonaccy(strange_fibonacci)`, and this will return the following function:
```python
def g(n):
        if n == 0 or n == 1:
            return 1
        else:
            return (
                strange_fibonacci(strange_fibonacci)(n-1) +
                strange_fibonacci(strange_fibonacci)(n-2)
            )
```
which, when called with a specific integer, will define new `g`s and call them in a nested way.

Let's test it with a specific number:

```python
recursive_function(strange_fibonacci)(10)
```

    89

Can you figure out _exactly_ what is happening here? It's a bit mindbending.

We can also get rid of the `f(f)` inside `strange_fibonacci`, since it still kind of feels like a type of recursion. To do that, we define the following function:

```python
apply_inside = lambda f: lambda g: f(lambda y: g(g)(y))
```

How does `apply_inside` work?

- You pass it some function $f$ ($f$ takes a function as argument), and it returns another function, let's call it $h$.
- $h$ (i.e., the return value of `apply_inside`) is defined as `lambda g: f(lambda y: g(g)(y)`, and so it is a function:
    - from a function $g$
    - to the result of calling $f$ (the original argument) on `lambda y: g(g)(y)`

This might become clearer with an example. Let's define another version of our strange fibonacci function, this time without the `f(f)` bit:

```python
def strange_fibonacci(f):
    # takes a function f and
    # defines a new function
    # whose behaviour depends on
    # the input function f
    def g(n):
        if n == 0 or n == 1:
            return 1
        else:
            return f(n-1) + f(n-2)
    return g
```

Now consider `apply_inside(strange_fibonacci)`. How does it work? Well, we know from the definition of `apply_inside` that it will have value:

`apply_inside(strange_fibonacci)` = `lambda g: strange_fibonacci(lambda y: g(g)(y))`

Note that now the argument of `strange_fibonacci` is `lambda y: g(g)(y)`, which is a function from a number to a function applied to itself and then to the number. Within `strange_fibonacci`, the input function appears in the line `return f(n-1) + f(n-2)`, which becomes:
- `return (lambda y: g(g)(y))(n-1) + (lambda y: g(g)(y))(n-2)`

and therefore:
- `return g(g)(n-1) + g(g)(n-2)`

And this is just a version of the `f(f)` bit we had at the beginning, which we have now successfully isolated into its own function (apply inside)!

Let's make sure this works:

```python
recursive_function(apply_inside(strange_fibonacci))(10)
```

    89

This section was a bit mental gymnastics to give you an idea of how messy things can get when higher-order functions are involved (i.e., function that take functions as arguments). We will see a lot of higher-order functions in the rest of the semester, so try to become comfortable with them!

## Plotting in Matplotlib

Since we'll be want to plot things in the second half of the course, it makes sense to introduce some simple cases now. We'll be using the most popular plotting library for python, `matplotlib`. We usually import it as follows:

```python
import matplotlib.pyplot as plt
```

Usually, we'll initialize a new plot with the following line:
```python
fig, axes = plt.subplots()
```
What `plt.subplots()` does is create a new figure object and a new `axes` object. The figure object is the whole figure, which can contain one or more axes. If you are unfamiliar with the `plt.subplots` interface, have a look [here](https://matplotlib.org/3.5.0/api/_as_gen/matplotlib.pyplot.subplots.html).

The axes are objects that have a bunch of methods, corresponding e.g. to different types of plots that can be called on them. Let's see an example of the `scatter` method of axes, which is used to make scatter plots:

```python
fig, axes = plt.subplots()

# create some new data to plot
# don't worry too much if you don't understand this line
x_scatter, y_scatter = np.random.rand(2,10)

# create a scatterplot with the given
# x and y coordinates
axes.scatter(x_scatter, y_scatter)

# show the plot
plt.show()
```

Another important function is to draw line plots, called simply `plot`:

```python
fig, axes = plt.subplots()

# create some new data to plot
# don't worry too much if you don't understand this line
x_line = y_line = np.arange(10)

# create a line plot with the given
# x and y coordinates
axes.plot(x_line, y_line)

# show the plot
plt.show()
```

Since we will be dealing with discrete variables a lot, it will be useful to also draw barplot, with the method `bar`:

```python
fig, axes = plt.subplots()

# create some new data to plot
# don't worry too much if you don't understand this line
x_bar = y_bar = np.arange(10)

# create a bar plot with the given
# x and height values
axes.bar(x_bar, y_bar)

# show the plot
plt.show()
```

We will also need to know what we are plotting, so we need to change the labels on the x-axis:

```python
fig, axes = plt.subplots()

# create some new data to plot
# don't worry too much if you don't understand this line
x_bar = y_bar = np.arange(10)

# create a bar plot with the given
# x and height values
axes.bar(x_bar, y_bar)

# define some labels for each x value
# don't worry if you don't understand this line
x_tick_labels = [f'lab-{x}' for x in range(len(x_bar))]

# tell plt that we'll specify one tick for each bar
axes.set_xticks(range(len(x_bar)))

# set the tick labels
axes.set_xticklabels(x_tick_labels)

# show the plot
plt.show()
```

## Classes, what's the idea?

We talked above about how we use functions because we don't want to rewrite the same stuff over and over again. But suppose that you don't want to just repeat a function, but a whole collection of functions and properties that can depend on each other. Then, you could define a class. In this light, a class is a _template for an object_, where an object has specific values for those variables that you left unspecified in the class.

Let's consider an example to make the explanation above clearer (as it often happens, the definitions are interdependent and the best way is to see how the terms are used in practice). Suppose we want to write a program that models students at a university. A very natural way of structuring this program is to have variables that correspond to students, e.g., one variable called `mark` for Mark, etc.

What type should these variables be? Clearly they're not integers, or strings, or floats. Ideally we'd want to have a _student_ type with its own methods that we can freely define. In other words, we want to have a `Student` class (Note: class names always start with a capital letter in pyhon by convention), which can produce specific `student` objects (`mark`, etc.). Here's how we can define it in python:

```python
# a class is defined with the keyword 'class'
# followed by the name of the class
# (remember: first letter capitalized!)
class Student:

    # __init__ is a special function
    # that gets called when a new object
    # is created from the class.
    # 'self' contains the properties
    # and methods defined in the class itself
    # and is passed to every function in the class
    def __init__(self, name, age):
        # define some class attributed
        # to store data about the student
        self.name = name
        self.age = age
        self.grades = []

    # define a new function that we can
    # use to record new grades that a student got.
    # This function takes a grade and
    # adds it to the student's record.
    def add_grade(self, grade):
        # The grades record is a list.
        # Append the new grade to the list!
        self.grades.append(grade)

    def average_grade(self):
        # Calculate the average of the grades
        # recorded in the grades list
        # and return the resulting mean.
        return np.mean(self.grades)
```

Now we can see the notation for instantiating a specific object (in this case, a student called 'Mark') from a class:

```python
# an object is defined by calling
# the class name with the arguments
# defined in the __init__ function
mark = Student('mark', 22)

# add some grade to Mark
mark.add_grade(1.3)
mark.add_grade(1.0)

# print Mark's average grade
print(mark.average_grade())
```

    1.15

You don't need to understand everything about classes, and you won't have to define one from scratch, but we'll have to adapt them as part of working with Piantadosi's LOTlib3 library. If you want to read more about classes in python [here is the documentation](https://docs.python.org/3/tutorial/classes.html) and here is [another explanation of the basic idea](https://towardsdatascience.com/introduction-to-python-classes-da526ff745df).

## Error handling

As you have already seen above, if something goes wrong sometimes python raises an _error_. For instance, above we have encountered `RecursionError: maximum recursion depth exceeded`. However, there are other types of errors in python, for instance if you try to get an element from an index that's too high for a list:

```python
a = [1, 3, 4]
print(a[3])
```

In general, errors help us understand what went wrong with our code. However, in python they can also be used as part of the program flow. Specifically, we might predict a situation that causes a specific error and write in the code what we want to happen is such a situation occurs. Here's how we do it:

```python
a = [1, 2, 4, 5]

# warn python that the indented code
# might contain an error
try:
    for i in range(10):
        print(a[i], end=', ')
# tell python what to do in case
# a specific kind of error occurs
# In this case, an IndexError
except IndexError:
    print('Finished: There are fewer than 10 elements in the list!')
```

    1, 2, 4, 5, Finished: There are fewer than 10 elements in the list!

Here's another error you should be familiar with (it will come up again later in the course!)

```python
1/0
```

And how we can deal with it:

```python
a = 0

try:
    print(5 / a)
except ZeroDivisionError:
    print('undefined')
```

    undefined

# First homeworks set

- Write a function to sum three numbers.
- Write a $\lambda$ function to sum three numbers.
- Write a recursive function that calculates the factorial function for $n$, written $n!$, and defined as follows:

\begin{align}
    1! &= 1 \\
    n! &= n \times (n-1)!
\end{align}

- Write a recursive function `interpret` that takes a string that consists of a series of `¬` followed by a single `a` (`a`, `¬a`, `¬¬a`, `¬¬¬a`, etc) and is defined as follows:

\begin{align}
    \text{interpret}(x) &= 1                             & \text{if } x = \text{'a'} \\
                        &= 1 - \text{interpret(x[1:])}   & \text{else}
\end{align}
    (do you see how this corresponds to the semantics of Boolean negation?)

- Rewrite the factorial function and `interpret` using `recursive_function` and `apply_inside` rather than explicit recursion.
- Make a scatterplot of the first 10 strings interpreted by `interpret`  (`a`, `¬a`, `¬¬a`, `¬¬¬a`, etc): the x-axis shows the strings in order and the y-axis shows the result of `interpret` for the corresponding string.
- Define a `Teacher` class that contains a record of the students that the teacher taught to, as well as the average grade the teacher gave to all the students (assume for simplicity that this is a school where each student only has one teacher). The class should have:
    - an `add_student` method that takes a Student object as defined above.
    - an `add_grade` method that takes a student name and a grade, and adds the grade to the student.
    - an `calculate_average_grade` method that returns the average grade that the teacher gave across all students.

    Usage example:

```python
maryWoolnoth = Teacher('Mary Woolnoth')
student1 = Student('Frank', 10)
student2 = Student('Bob', 9)

maryWoolnoth.add_student(student1)
maryWoolnoth.add_student(student2)

maryWoolnoth.add_grade('Frank', 1.3)
```
- Google how to read a text file and print out its content. Write code that does this, but using the `try ... except ...` statements prints 'File does not exist!' if the file does not exist.


```python
%matplotlib inline
```

# Fit a banded ridge model with both wordnet and motion energy features

In this example, we model the fMRI responses with a `banded ridge regression`,
with two different feature spaces: motion energy and wordnet categories.

*Banded ridge regression:* Since the relative scaling of both feature spaces is
unknown, we use two regularization hyperparameters (one per feature space) in a
model called banded ridge regression [1]_. Just like with ridge regression, we
optimize the hyperparameters over cross-validation. An efficient implementation
of this model is available in the `himalaya
<https://github.com/gallantlab/himalaya>`_ package.

*Running time:* This example is more computationally intensive than the
previous examples. With a GPU backend, model fitting takes around 6 minutes.
With a CPU backend, it can last 10 times more.

```python

```

## Path of the data directory

```python
import os
from voxelwise_tutorials.io import get_data_home
directory = os.path.join(get_data_home(), "vim-5")
print(directory)
```

```python
# modify to use another subject
subject = "S01"
```

## Load the data

As in the previous examples, we first load the fMRI responses, which are our
regression targets.

```python
import numpy as np

from voxelwise_tutorials.io import load_hdf5_array

file_name = os.path.join(directory, "responses", f"{subject}_responses.hdf")
Y_train = load_hdf5_array(file_name, key="Y_train")
Y_test = load_hdf5_array(file_name, key="Y_test")

print("(n_samples_train, n_voxels) =", Y_train.shape)
print("(n_repeats, n_samples_test, n_voxels) =", Y_test.shape)
```

We also compute the explainable variance, to exclude voxels with low
explainable variance from the fit, and speed up the model fitting.

```python
from voxelwise_tutorials.utils import explainable_variance
ev = explainable_variance(Y_test)
print("(n_voxels,) =", ev.shape)

mask = ev > 0.1
print("(n_voxels_mask,) =", ev[mask].shape)
```

We average the test repeats, to remove the non-repeatable part of fMRI
responses.

```python
Y_test = Y_test.mean(0)

print("(n_samples_test, n_voxels) =", Y_test.shape)
```

We fill potential NaN (not-a-number) values with zeros.

```python
Y_train = np.nan_to_num(Y_train)
Y_test = np.nan_to_num(Y_test)
```

And we make sure the targets are centered.

```python
Y_train -= Y_train.mean(0)
Y_test -= Y_test.mean(0)
```

Then we load both feature spaces, that are going to be used for the
linear regression model.

```python
feature_names = ["wordnet", "motion_energy"]

Xs_train = []
Xs_test = []
n_features_list = []
for feature_space in feature_names:
    file_name = os.path.join(directory, "features", f"{feature_space}.hdf")
    Xi_train = load_hdf5_array(file_name, key="X_train")
    Xi_test = load_hdf5_array(file_name, key="X_test")

    Xs_train.append(Xi_train.astype(dtype="float32"))
    Xs_test.append(Xi_test.astype(dtype="float32"))
    n_features_list.append(Xi_train.shape[1])

# concatenate the feature spaces
X_train = np.concatenate(Xs_train, 1)
X_test = np.concatenate(Xs_test, 1)

print("(n_samples_train, n_features_total) =", X_train.shape)
print("(n_samples_test, n_features_total) =", X_test.shape)
print("[n_features_wordnet, n_features_motion_energy] =", n_features_list)
```

## Define the cross-validation scheme

We define again a leave-one-run-out cross-validation split scheme.

```python
from sklearn.model_selection import check_cv
from voxelwise_tutorials.utils import generate_leave_one_run_out

# indice of first sample of each run
run_onsets = load_hdf5_array(file_name, key="run_onsets")
print(run_onsets)
```

We define a cross-validation splitter, compatible with ``scikit-learn`` API.

```python
n_samples_train = X_train.shape[0]
cv = generate_leave_one_run_out(n_samples_train, run_onsets)
cv = check_cv(cv)  # copy the cross-validation splitter into a reusable list
```

## Define the model

The model pipeline contains similar steps than the pipeline from previous
examples. We remove the mean of each feature with a ``StandardScaler``,
and add delays with a ``Delayer``.

```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from voxelwise_tutorials.delayer import Delayer
from himalaya.backend import set_backend
backend = set_backend("torch_cuda", on_error="warn")
```

To fit the banded ridge model, we use ``himalaya``'s
``MultipleKernelRidgeCV`` model, with a separate linear kernel per feature
space. Similarly to ``KernelRidgeCV``, the model optimizes the
hyperparameters over cross-validation. However, while ``KernelRidgeCV`` has
to optimize only one hyperparameter (``alpha``), ``MultipleKernelRidgeCV``
has to optimize ``m`` hyperparameters, where ``m`` is the number of feature
spaces (here ``m = 2``). To do so, the model implements two different
solvers, one using hyperparameter random search, and one using hyperparameter
gradient descent. For large number of targets, we recommend using the
random-search solver.

The class takes a number of common parameters during initialization, such as
``kernels``, or ``solver``. Since the solver parameters vary depending on the
solver used, they are passed as a ``solver_params`` dictionary.

```python
from himalaya.kernel_ridge import MultipleKernelRidgeCV

# Here we will use the "random_search" solver.
solver = "random_search"

# We can check its specific parameters in the function docstring:
solver_function = MultipleKernelRidgeCV.ALL_SOLVERS[solver]
print("Docstring of the function %s:" % solver_function.__name__)
print(solver_function.__doc__)
```

The hyperparameter random-search solver separates the hyperparameters into a
shared regularization ``alpha`` and a vector of positive kernel weights which
sum to one. This separation of hyperparameters allows to explore efficiently
a large grid of values for ``alpha`` for each sampled kernel weights vector.

We use *20* random-search iterations to have a reasonably fast example. To
have better results, especially for larger number of feature spaces, one
might need more iterations. (Note that there is currently no stopping
criterion in the random-search method.)

```python
n_iter = 20

alphas = np.logspace(1, 20, 20)
```

Batch parameters, used to reduce the necessary GPU memory. A larger value
will be a bit faster, but the solver might crash if it is out of memory.
Optimal values depend on the size of your dataset.

```python
n_targets_batch = 200
n_alphas_batch = 5
n_targets_batch_refit = 200
```

We put all these parameters in a dictionary ``solver_params``, and define
the main estimator ``MultipleKernelRidgeCV``.

```python
solver_params = dict(n_iter=n_iter, alphas=alphas,
                     n_targets_batch=n_targets_batch,
                     n_alphas_batch=n_alphas_batch,
                     n_targets_batch_refit=n_targets_batch_refit)

mkr_model = MultipleKernelRidgeCV(kernels="precomputed", solver=solver,
                                  solver_params=solver_params, cv=cv)
```

We need a bit more work than in previous examples before defining the full
pipeline, since the banded ridge model requires `multiple` precomputed
kernels, one for each feature space. To compute them, we use the
``ColumnKernelizer``, which can create multiple kernels from different
column of your features array. ``ColumnKernelizer`` works similarly to
``scikit-learn``'s ``ColumnTransformer``, but instead of returning a
concatenation of transformed features, it returns a stack of kernels,
as required in ``MultipleKernelRidgeCV(kernels="precomputed")``.

First, we create a different ``Kernelizer`` for each feature space.
Here we use a linear kernel for all feature spaces, but ``ColumnKernelizer``
accepts any ``Kernelizer``, or ``scikit-learn`` ``Pipeline`` ending with a
``Kernelizer``.

```python
from himalaya.kernel_ridge import Kernelizer
from sklearn import set_config
set_config(display='diagram')  # requires scikit-learn 0.23

preprocess_pipeline = make_pipeline(
    StandardScaler(with_mean=True, with_std=False),
    Delayer(delays=[1, 2, 3, 4]),
    Kernelizer(kernel="linear"),
)
preprocess_pipeline
```

The column kernelizer applies a different pipeline on each selection of
features, here defined with ``slices``.

```python
from himalaya.kernel_ridge import ColumnKernelizer

# Find the start and end of each feature space in the concatenated ``X_train``.
start_and_end = np.concatenate([[0], np.cumsum(n_features_list)])
slices = [
    slice(start, end)
    for start, end in zip(start_and_end[:-1], start_and_end[1:])
]
slices
```

```python
kernelizers_tuples = [(name, preprocess_pipeline, slice_)
                      for name, slice_ in zip(feature_names, slices)]
column_kernelizer = ColumnKernelizer(kernelizers_tuples)
column_kernelizer

# (Note that ``ColumnKernelizer`` has a parameter ``n_jobs`` to parallelize
# each ``Kernelizer``, yet such parallelism does not work with GPU arrays.)
```

Then we can define the model pipeline.

```python
pipeline = make_pipeline(
    column_kernelizer,
    mkr_model,
)
pipeline
```

## Fit the model

We fit on the train set, and score on the test set.

To speed up the fit and to limit the memory peak in Colab, we only fit on
voxels with explainable variance above 0.1.

With a GPU backend, the fitting of this model takes around 6 minutes. With a
CPU backend, it can last 10 times more.

```python
pipeline.fit(X_train, Y_train[:, mask])

scores_mask = pipeline.score(X_test, Y_test[:, mask])
scores_mask = backend.to_numpy(scores_mask)
print("(n_voxels_mask,) =", scores_mask.shape)

# Then we extend the scores to all voxels, giving a score of zero to unfitted
# voxels.
n_voxels = Y_train.shape[1]
scores = np.zeros(n_voxels)
scores[mask] = scores_mask
print("(n_voxels,) =", scores.shape)
```

## Compare with a ridge model

We can compare with a baseline model, which does not use one hyperparameter
per feature space, but instead shares the same hyperparameter for all feature
spaces.

```python
from himalaya.kernel_ridge import KernelRidgeCV

pipeline_baseline = make_pipeline(
    StandardScaler(with_mean=True, with_std=False),
    Delayer(delays=[1, 2, 3, 4]),
    KernelRidgeCV(
        alphas=alphas, cv=cv,
        solver_params=dict(n_targets_batch=n_targets_batch,
                           n_alphas_batch=n_alphas_batch,
                           n_targets_batch_refit=n_targets_batch_refit)),
)
pipeline_baseline
```

```python
pipeline_baseline.fit(X_train, Y_train[:, mask])
scores_baseline_mask = pipeline_baseline.score(X_test, Y_test[:, mask])
scores_baseline_mask = backend.to_numpy(scores_baseline_mask)

# extend to unfitted voxels
n_voxels = Y_train.shape[1]
scores_baseline = np.zeros(n_voxels)
scores_baseline[mask] = scores_baseline_mask
```

Here we plot the comparison of model prediction accuracies with a 2D
histogram. All 70k voxels are represented in this histogram, where the
diagonal corresponds to identical model prediction accuracy for both models.
A distibution deviating from the diagonal means that one model has better
predictive performance than the other.

```python
import matplotlib.pyplot as plt
from voxelwise_tutorials.viz import plot_hist2d

ax = plot_hist2d(scores_baseline, scores)
ax.set(title='Generalization R2 scores', xlabel='KernelRidgeCV',
       ylabel='MultipleKernelRidgeCV')
plt.show()
```

We see that the banded ridge model (``MultipleKernelRidgeCV``) outperforms
the ridge model (``KernelRidegeCV``). Indeed, banded ridge regression is able
to find the optimal scalings of each feature space, independently on each
voxel. Banded ridge regression is thus able to perform a soft selection
between the available feature spaces, based on the cross-validation
performances.

## Plot the banded ridge split

On top of better prediction accuracy, banded ridge regression also gives a
way to disentangle the contribution of the two feature spaces. To do so, we
take the kernel weights and the ridge (dual) weights corresponding to each
feature space, and use them to compute the prediction from each feature space
separately.

\begin{align}\hat{y} = \sum_i^m \hat{y}_i = \sum_i^m \gamma_i K_i \hat{w}\end{align}

Then, we use these split predictions to compute split $\tilde{R}^2_i$
scores. These scores are corrected so that their sum is equal to the
$R^2$ score of the full prediction $\hat{y}$.

```python
from himalaya.scoring import r2_score_split

Y_test_pred_split = pipeline.predict(X_test, split=True)
split_scores_mask = r2_score_split(Y_test[:, mask], Y_test_pred_split)

print("(n_kernels, n_samples_test, n_voxels_mask) =", Y_test_pred_split.shape)
print("(n_kernels, n_voxels_mask) =", split_scores_mask.shape)

# extend to unfitted voxels
n_kernels = split_scores_mask.shape[0]
n_voxels = Y_train.shape[1]
split_scores = np.zeros((n_kernels, n_voxels))
split_scores[:, mask] = backend.to_numpy(split_scores_mask)
print("(n_kernels, n_voxels) =", split_scores.shape)
```

We can then plot the split scores on a flatmap with a 2D colormap.

```python
from voxelwise_tutorials.viz import plot_2d_flatmap_from_mapper

mapper_file = os.path.join(directory, "mappers", f"{subject}_mappers.hdf")
ax = plot_2d_flatmap_from_mapper(split_scores[0], split_scores[1],
                                 mapper_file, vmin=0, vmax=0.25, vmin2=0,
                                 vmax2=0.5, label_1=feature_names[0],
                                 label_2=feature_names[1])
plt.show()
```

The blue regions are better predicted by the motion-energy features, the
orange regions are better predicted by the wordnet features, and the white
regions are well predicted by both feature spaces.

Compared to the last figure of the previous example, we see that most white
regions have been replaced by either blue or orange regions. The banded ridge
regression disentangled the two feature spaces in voxels where both feature
spaces had good prediction accuracy (see previous example). For example,
motion-energy features predict brain activity in early visual cortex, while
wordnet features predict in semantic visual areas. For more discussions about
these results, we refer the reader to the original publication [1]_.

## References

.. [1] Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019).
    Voxelwise encoding models with non-spherical multivariate normal priors.
    Neuroimage, 197, 482-492.


# Simulated Satellite Scan Strategies

```python
# Load common tools for all lessons
import sys
sys.path.insert(0, "..")
from lesson_tools import (
    fake_focalplane
)
```

## Overview

A generic satellite scanning strategy can be described in terms of precession and spin axes, the opening angles about each of them, and the rates of rotation.  The precession axis itself is typically oriented in the anti-sun direction and that orientation must be updated either in steps or with a continuous slewing motion.  Here is a cartoon (drawn from an old, public LiteBIRD talk) showing a sketch of these angles:

For a real satellite experiment such as Planck or LiteBIRD, there are many custom details, such as simulating repointing maneuvers, simulating any lost time due to cooler cycling, nutation effects, etc.  Those are contained in classes for the specific experiments.  The tools built in to the core TOAST package are intended for rough simulations to study things like scan strategy choices.

### TOD Class for Simulations

In the introductory lesson we saw the use of a TOD derived class providing things like telescope pointing.  Here we introduce the `TODSatellite` class which serves that purpose for generic satellite simulations.

```python
import numpy as np

import toast
from toast.todmap import (
    slew_precession_axis,
    TODSatellite
)
```

```python
# Default Comm (one group for this example)

comm = toast.Comm()
```

```python
# Create our fake focalplane

fp = fake_focalplane()

detnames = list(sorted(fp.keys()))
detquat = {x: fp[x]["quat"] for x in detnames}
```

```python
# Scan parameters (made up, not physically motivated)

samplerate = 10.0
precperiod = 90.0
precangle = 45.0
spinperiod = 1.0
spinangle = 45.0
```

```python
# We can simulate a simplistic HWP

hwprpm = 6.0
```

```python
# Number of samples

nsamples = 100
```

```python
# Instantiate a TOD

tod = TODSatellite(
    comm.comm_group,
    detquat,
    nsamples,
    firstsamp=0,
    firsttime=0.0,
    rate=samplerate,
    spinperiod=spinperiod,
    spinangle=spinangle,
    precperiod=precperiod,
    precangle=precangle,
    coord="E",
    hwprpm=hwprpm
)
```

```python
# The TOD constructor above specifies the scan parameters, but the boresight
# simulation is not done until we set the location of the precession axis as a
# function of time.  There is a reason for this delayed construction.  The data
# distribution occurs during the above construction, and we might want to only
# simulate the precession axis motion for our local data.  In reality this is
# cheap enough to do on one process and distribute during the construction.

qprec = np.empty(4 * tod.local_samples[1], dtype=np.float64).reshape((-1, 4))

deg_per_day = 1.0

slew_precession_axis(
    qprec,
    firstsamp=tod.local_samples[0],
    samplerate=samplerate,
    degday=deg_per_day,
)

tod.set_prec_axis(qprec=qprec)
```

```python
# Now we can read from this TOD object

print("TOD timestampes = {} ...".format(tod.read_times()[:5]))
print("TOD boresight = \n{} ...".format(tod.read_boresight()[:5,:]))
for d in detnames:
    print("TOD detector {} = {} ...".format(d, tod.read(detector=d, n=5)))
    print("TOD detector {} flags = {} ...".format(d, tod.read_flags(detector=d, n=5)))
    print("TOD detector {} pointing = {} ...".format(d, tod.read_pntg(detector=d, n=5)))
```

    TOD timestampes = [0.  0.1 0.2 0.3 0.4] ...
    TOD boresight =
    [[ 0.5         0.5        -0.5         0.5       ]
     [ 0.50372461  0.50002202 -0.49626167  0.49996385]
     [ 0.50743541  0.50002989 -0.49250974  0.49991357]
     [ 0.51113228  0.50002363 -0.4887443   0.49984914]
     [ 0.51481514  0.50000321 -0.48496547  0.49977059]] ...
    TOD detector 0A = [0. 0. 0. 0. 0.] ...
    TOD detector 0A flags = [0 0 0 0 0] ...
    TOD detector 0A pointing = [[ 0.65328148  0.27059805 -0.27059805  0.65328148]
     [ 0.656731    0.26919305 -0.26715812  0.65181749]
     [ 0.66016234  0.26778026 -0.26371103  0.65033523]
     [ 0.66357541  0.26635974 -0.26025687  0.64883474]
     [ 0.66697012  0.26493151 -0.25679575  0.64731607]] ...
    TOD detector 0B = [0. 0. 0. 0. 0.] ...
    TOD detector 0B flags = [0 0 0 0 0] ...
    TOD detector 0B pointing = [[ 0.65328148 -0.27059805  0.27059805  0.65328148]
     [ 0.65472717 -0.27403071  0.27199525  0.64981388]
     [ 0.65615451 -0.27745603  0.27338459  0.64632831]
     [ 0.65756345 -0.2808739   0.27476605  0.64282484]
     [ 0.65895396 -0.28428423  0.27613957  0.6393036 ]] ...
    TOD detector 1A = [0. 0. 0. 0. 0.] ...
    TOD detector 1A flags = [0 0 0 0 0] ...
    TOD detector 1A pointing = [[ 0.50501286  0.50501286 -0.49493637  0.49493637]
     [ 0.50869961  0.50503451 -0.4911607   0.4949    ]
     [ 0.51237241  0.50504188 -0.48737156  0.49484964]
     [ 0.51603116  0.50503497 -0.48356907  0.49478528]
     [ 0.51967576  0.50501376 -0.47975332  0.49470694]] ...
    TOD detector 1B = [0. 0. 0. 0. 0.] ...
    TOD detector 1B flags = [0 0 0 0 0] ...
    TOD detector 1B pointing = [[ 7.14196038e-01 -5.55111512e-17 -1.11022302e-16  6.99945726e-01]
     [ 7.16818276e-01 -2.59161549e-03  2.64408538e-03  6.97250208e-01]
     [ 7.19420549e-01 -5.18346747e-03  5.28779671e-03  6.94535272e-01]
     [ 7.22002785e-01 -7.77548483e-03  7.93106150e-03  6.91800997e-01]
     [ 7.24564909e-01 -1.03675965e-02  1.05738073e-02  6.89047458e-01]] ...
    TOD detector 2A = [0. 0. 0. 0. 0.] ...
    TOD detector 2A flags = [0 0 0 0 0] ...
    TOD detector 2A pointing = [[ 0.65417834  0.27764851 -0.26352011  0.6523183 ]
     [ 0.65759801  0.27622036 -0.26005044  0.65087705]
     [ 0.6609995   0.27478423 -0.25657382  0.64941756]
     [ 0.66438269  0.27334016 -0.25309032  0.64793988]
     [ 0.6677475   0.2718882  -0.24960006  0.64644404]] ...
    TOD detector 2B = [0. 0. 0. 0. 0.] ...
    TOD detector 2B flags = [0 0 0 0 0] ...
    TOD detector 2B pointing = [[ 0.65890108 -0.26624679  0.27492183  0.64759555]
     [ 0.6603093  -0.26967473  0.27635614  0.64412301]
     [ 0.66169902 -0.27309543  0.27778248  0.64063265]
     [ 0.66307019 -0.27650882  0.2792008   0.63712456]
     [ 0.66442277 -0.27991479  0.28061107  0.63359886]] ...
    TOD detector 3A = [0. 0. 0. 0. 0.] ...
    TOD detector 3A flags = [0 0 0 0 0] ...
    TOD detector 3A pointing = [[ 0.49309224  0.50181874 -0.49813049  0.50685699]
     [ 0.49683581  0.50180832 -0.49441092  0.50685345]
     [ 0.50056576  0.50178371 -0.49067781  0.50683559]
     [ 0.50428199  0.5017449  -0.48693124  0.50680341]
     [ 0.50798438  0.50169191 -0.48317134  0.50675689]] ...
    TOD detector 3B = [0. 0. 0. 0. 0.] ...
    TOD detector 3B flags = [0 0 0 0 0] ...
    TOD detector 3B pointing = [[ 0.7035083   0.00617057  0.00617057  0.71063346]
     [ 0.70614804  0.00351609  0.0087982   0.70800083]
     [ 0.70876811  0.00086122  0.01142528  0.70534849]
     [ 0.71136844 -0.00179399  0.01405174  0.70267651]
     [ 0.71394896 -0.00444945  0.01667751  0.69998497]] ...
    TOD detector 4A = [0. 0. 0. 0. 0.] ...
    TOD detector 4A flags = [0 0 0 0 0] ...
    TOD detector 4A pointing = [[ 0.49493637  0.49493637 -0.50501286  0.50501286]
     [ 0.49869846  0.49495875 -0.50131225  0.50497694]
     [ 0.50244687  0.49496713 -0.4975979   0.50492673]
     [ 0.50618151  0.49496151 -0.49386991  0.50486225]
     [ 0.50990226  0.49494189 -0.49012838  0.5047835 ]] ...
    TOD detector 4B = [0. 0. 0. 0. 0.] ...
    TOD detector 4B flags = [0 0 0 0 0] ...
    TOD detector 4B pointing = [[ 6.99945726e-01  0.00000000e+00 -1.11022302e-16  7.14196038e-01]
     [ 7.02621751e-01 -2.64437272e-03  2.59132230e-03  7.11553911e-01]
     [ 7.05278207e-01 -5.28897433e-03  5.18226587e-03  7.08891968e-01]
     [ 7.07915019e-01 -7.93373230e-03  7.77275966e-03  7.06210285e-01]
     [ 7.10532113e-01 -1.05785741e-02  1.03627326e-02  7.03508937e-01]] ...
    TOD detector 5A = [0. 0. 0. 0. 0.] ...
    TOD detector 5A flags = [0 0 0 0 0] ...
    TOD detector 5A pointing = [[ 0.50181874  0.49309224 -0.50685699  0.49813049]
     [ 0.50556168  0.49314706 -0.50313781  0.49806195]
     [ 0.50929075  0.49318794 -0.49940483  0.49797932]
     [ 0.51300585  0.49321485 -0.49565816  0.49788261]
     [ 0.51670688  0.49322781 -0.49189789  0.49777183]] ...
    TOD detector 5B = [0. 0. 0. 0. 0.] ...
    TOD detector 5B flags = [0 0 0 0 0] ...
    TOD detector 5B pointing = [[ 0.7035083  -0.00617057 -0.00617057  0.71063346]
     [ 0.70619373 -0.00877846 -0.00358917  0.70795514]
     [ 0.70885948 -0.01138641 -0.00100798  0.7052571 ]
     [ 0.71150548 -0.01399435  0.00157293  0.70253942]
     [ 0.71413167 -0.01660221  0.0041535   0.69980217]] ...
    TOD detector 6A = [0. 0. 0. 0. 0.] ...
    TOD detector 6A flags = [0 0 0 0 0] ...
    TOD detector 6A pointing = [[ 0.65890108  0.26624679 -0.27492183  0.64759555]
     [ 0.66234515  0.26487916 -0.2714774   0.64609439]
     [ 0.66577088  0.26350386 -0.26802568  0.64457512]
     [ 0.66917818  0.26212094 -0.26456677  0.64303779]
     [ 0.67256696  0.26073044 -0.26110078  0.64148243]] ...
    TOD detector 6B = [0. 0. 0. 0. 0.] ...
    TOD detector 6B flags = [0 0 0 0 0] ...
    TOD detector 6B pointing = [[ 0.65417834 -0.27764851  0.26352011  0.6523183 ]
     [ 0.65564659 -0.28105089  0.26489422  0.64882123]
     [ 0.65709647 -0.28444573  0.26626067  0.64530622]
     [ 0.65852793 -0.28783293  0.26761942  0.64177334]
     [ 0.65994092 -0.2912124   0.26897045  0.63822271]] ...

Notice that the signal data for all detectors is zero.  For simulated TOD classes, there is no data to "read".  Instead, simulated timestreams are constructed and stored in the `tod.cache` member variable.

### Low Resolution Example

Imagine the case of a satellite telescope with detector beams that are a 5 degrees FWHM.  We'll use a healpix resolution of NSIDE = 32 (approximately 2 degrees) for this example.  Let's use made-up angles for the spin and precession angles of 40 and 50 degrees, respectively:

\begin{align}
\alpha & = 50^{\circ}\\
\beta & = 40^{\circ}\\
\omega_{\alpha} & = \text{precession rate}\\
\omega_{\beta} & = \text{spin rate}\\
\end{align}

When computing the precession rate, we want the precession motion to be slow enough so that the speed of the boresight on the sky does not vary enough to change our effective beams.  The speed variation on the sky due to precession is

\begin{align}
v_{\text{min}} & = \beta \cdot \omega_{\beta} - \alpha \cdot \omega_{\alpha}\\
v_{\text{max}} & = \beta \cdot \omega_{\beta} + \alpha \cdot \omega_{\alpha}\\
v_{\text{diff}} & = v_{\text{max}} - v_{\text{min}} = 2 \alpha \omega_{\alpha}
\end{align}

This change, integrated over a sample, must be a small fraction (here called "$X$") of the beam FWHM:

\begin{align}
\frac{2 \alpha \omega_{\alpha}}{f_{\text{sample}}} & = X \cdot \text{FWHM}\\
f_{\text{sample}} & = \frac{2 \alpha \omega_{\alpha}}{X \cdot \text{FWHM}}\\
\end{align}

The speed of the boresight on the sky in degrees per second due to the spin axis motion is

$$v_{bore} = \alpha \cdot \omega_{\alpha} \cdot \frac{1}{60}$$

If we want to have 3 hits per pixel with two degree pixels (2/3 degree per second), this gives us

\begin{align}
v_{bore} = \frac{2}{3} & = \alpha \cdot \omega_{\alpha} \cdot \frac{1}{60}\\
\omega_{\alpha} & = \frac{60}{3 \cdot \alpha} = 0.8\;\text{RPM}
\end{align}

If we assume five percent for our "$X$" fraction above, then this in turn forces our sample rate to be:

$$f_{\text{sample}} = \frac{2 \cdot 50 \cdot 0.8}{0.05 \cdot 3.0 \cdot 60} = 8.9\;\text{Hz}$$

The precession rate is slower than the spin rate.  The spin rate above corresponds to a period of 1.25 minutes.  We choose a precession period 20 times longer (25 minutes).  We will assume a very simple satellite motion where the precession axis slews continuously in the anti-sun direction.

**NOTE:  For the serial example in the next cell, we have artificially decreased the sample rate to 0.5 Hz and the resolution to NSIDE=16.  This is so that this small example fits into reasonable RAM while still covering the sky.  See the parallel notebook for an example with proper sampling.**

```python
# Scan parameters

alpha = 50.0      # precession opening angle, degrees
beta = 45.0       # spin opening angle, degrees
p_alpha = 25.0    # precession period, minutes
p_beta = 1.25     # spin period, minutes
samplerate = 0.5  # sample rate, Hz
hwprpm = 5.0      # HWP rotation in RPM
nside = 16        # Healpix NSIDE
```

```python
# We will use one observation per day, with no gaps in between, and
# run for one year.

obs_samples = int(24 * 3600.0 * samplerate) - 1
nobs = 366
```

```python
# Slew the precession axis so that it completes one circle

deg_per_day = 360.0 / nobs
```

```python
# Create distributed data

data = toast.Data(comm)
```

```python
# Append observations

for ob in range(nobs):
    obsname = "{:03d}".format(ob)
    obsfirst = ob * (obs_samples + 1)
    obsstart = 24 * 3600.0
    tod = TODSatellite(
        comm.comm_group,
        detquat,
        obs_samples,
        firstsamp=obsfirst,
        firsttime=obsstart,
        rate=samplerate,
        spinperiod=p_beta,
        spinangle=beta,
        precperiod=p_alpha,
        precangle=alpha,
        coord="E",
        hwprpm=hwprpm
    )
    qprec = np.empty(4 * tod.local_samples[1], dtype=np.float64).reshape((-1, 4))
    slew_precession_axis(
        qprec,
        firstsamp=obsfirst,
        samplerate=samplerate,
        degday=deg_per_day,
    )
    tod.set_prec_axis(qprec=qprec)
    obs = dict()
    obs["tod"] = tod
    data.obs.append(obs)
```

Now that we have simulated our scan strategy, we can make a simple hit map to visualize this

```python
from toast.todmap import (
    get_submaps_nested,
    OpPointingHpix,
    OpAccumDiag
)
from toast.map import (
    DistPixels
)
```

```python
# Make a simple pointing matrix

pointing = OpPointingHpix(nside=nside, nest=True, mode="IQU")
pointing.exec(data)
```

```python
# Compute the locally hit pixels
#eg each process doesn't hold the whole sky, just the pixels it hits

localpix, localsm, subnpix = get_submaps_nested(data, nside)
```

```python
# Construct a distributed map to store the hit map

npix = 12 * nside**2

hits = DistPixels(
    comm=data.comm.comm_world,
    size=npix,
    nnz=1,
    dtype=np.int64,
    submap=subnpix,
    local=localsm,
)
hits.data.fill(0)
```

```python
# Accumulate the hit map locally

build_hits = OpAccumDiag(hits=hits)
build_hits.exec(data)
```

```python
# Reduce the map across processes (a No-op in this case)

hits.allreduce()
```

```python
# Write out the map

hitsfile = "simscan_satellite_hits.fits"
hits.write_healpix_fits(hitsfile)
```

```python
# Plot the map.  If we were running on multiple processes, then
# only rank zero would do this...

import healpy as hp
import matplotlib.pyplot as plt
%matplotlib inline

hitdata = hp.read_map(hitsfile, nest=True)
hp.mollview(hitdata, xsize=800, nest=True, cmap="cool", min=0)
plt.show()

```

```python

```


Lambda School Data Science

*Unit 2, Sprint 1, Module 2*

---

# Regression 2
- Do train/test split
- Use scikit-learn to fit a multiple regression
- Understand how ordinary least squares regression minimizes the sum of squared errors
- Define overfitting/underfitting and the bias/variance tradeoff

### Setup

Run the code cell below. You can work locally (follow the [local setup instructions](https://lambdaschool.github.io/ds/unit2/local/)) or on Colab.

Libraries:
- matplotlib
- numpy
- pandas
- plotly
- scikit-learn

```python
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Applied-Modeling/master/data/'

# If you're working locally:
else:
    DATA_PATH = '../data/'

# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
```

# Do train/test split

## Overview

### Predict Elections! 🇺🇸🗳

How could we try to predict the 2020 US Presidential election?

According to Douglas Hibbs, a political science and economics professor, you can [explain elections with just two features, "Bread and Peace":](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

> Aggregate two-party vote shares going to candidates of the party holding the presidency during the postwar era are well explained by just two fundamental determinants:
>
> (1) Positively by weighted-average growth of per capita real disposable personal income over the term.
> (2) Negatively by cumulative US military fatalities (scaled to population) owing to unprovoked, hostile deployments of American armed forces in foreign wars.

Let's look at the data that Hibbs collected and analyzed:

```python
import pandas as pd
df = pd.read_csv(DATA_PATH+'elections/bread_peace_voting.csv')
df
```

Data Sources & Definitions

- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

> Fatalities denotes the cumulative number of American military fatalities per millions of US population the in Korea, Vietnam, Iraq and Afghanistan wars during the presidential terms preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and 2012 elections. —[Hibbs](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 33

Here we have data from the 1952-2016 elections. We could make a model to predict 1952-2016 election outcomes — but do we really care about that?

No, not really. We already know what happened, we don't need to predict it.

This is explained in [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/), Chapter 2.2, Assessing Model Accuracy:

> In general, we do not really care how well the method works training on the training data. Rather, _we are interested in the accuracy of the predictions that we obtain when we apply our method to previously unseen test data._ Why is this what we care about?
>
> Suppose that we are interested in developing an algorithm to predict a stock’s price based on previous stock returns. We can train the method using stock returns from the past 6 months. But we don’t really care how well our method predicts last week’s stock price. We instead care about how well it will predict tomorrow’s price or next month’s price.
>
> On a similar note, suppose that we have clinical measurements (e.g. weight, blood pressure, height, age, family history of disease) for a number of patients, as well as information about whether each patient has diabetes. We can use these patients to train a statistical learning method to predict risk of diabetes based on clinical measurements. In practice, we want this method to accurately predict diabetes risk for _future patients_ based on their clinical measurements. We are not very interested in whether or not the method accurately predicts diabetes risk for patients used to train the model, since we already know which of those patients have diabetes.

So, we're really interested in the 2020 election — but we probably don't want to wait until then to evaluate our model.

There is a way we can estimate now how well our model will generalize in the future. We can't fast-forward time, but we can rewind it...

We can split our data in **two sets.** For example:
1. **Train** a model on elections before 2008.
2. **Test** the model on 2008, 2012, 2016.

This "backtesting" helps us estimate how well the model will predict the next elections going forward, starting in 2020.

This is explained in [_Forecasting,_ Chapter 3.4,](https://otexts.com/fpp2/accuracy.html) Evaluating forecast accuracy:

> The accuracy of forecasts can only be determined by considering how well a model performs on new data that were not used when fitting the model.
>
>When choosing models, it is common practice to separate the available data into two portions, training and test data, where the training data is used to estimate any parameters of a forecasting method and the test data is used to evaluate its accuracy. Because the test data is not used in determining the forecasts, it should provide a reliable indication of how well the model is likely to forecast on new data.
>
>
>
>The size of the test set is typically about 20% of the total sample, although this value depends on how long the sample is and how far ahead you want to forecast. The following points should be noted.
>
>- A model which fits the training data well will not necessarily forecast well.
>- A perfect fit can always be obtained by using a model with enough parameters.
>- Over-fitting a model to data is just as bad as failing to identify a systematic pattern in the data.
>
>Some references describe the test set as the “hold-out set” because these data are “held out” of the data used for fitting. Other references call the training set the “in-sample data” and the test set the “out-of-sample data”. We prefer to use “training data” and “test data” in this book.

**How should we split: Randomly? Before/after a given date?**

I recommend you all read a great blog post, [How (and why) to create a good validation set](https://www.fast.ai/2017/11/13/validation-sets/), by fast.ai cofounder Rachel Thomas.

She gives great examples to answer the question “When is a random subset not good enough?” I’m not as opposed to random splits as Rachel Thomas seems to be. But it’s worth thinking about the trade-offs!

Time-based and random splits can both be useful, and you’ll get repeated hands-on practice with both during this unit! (She also talks about the distinction between validation & test sets, which we’ll introduce in the last lesson of this Sprint.)

## Follow Along

Split the data in two sets:
1. Train on elections before 2008.
2. Test on 2008 and after.

```python

```

How many observations (rows) are in the train set? In the test set?

```python

```

Note that this volume of data is at least two orders of magnitude smaller than we usually want to work with for predictive modeling.

There are other validation techniques that could be used here, such as [time series cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split), or [leave-one-out cross validation](https://scikit-learn.org/stable/modules/cross_validation.html#leave-one-out-loo) for small datasets. However, for this module, let's start simpler, with train/test split.

Using a tiny dataset is intentional here. It's good for learning because we can see all the data at once.

## Challenge

In your assignment, you will do train/test split, based on date.

# Use scikit-learn to fit a multiple regression

## Overview

We've done train/test split, and we're ready to fit a model.

We'll proceed in 3 steps. The first 2 are review from the previous module. The 3rd is new.

- Begin with baselines (0 features)
- Simple regression (1 feature)
- Multiple regression (2 features)

## Follow Along

### Begin with baselines (0 features)

What was the average Incumbent Party Vote Share, in the 1952-2004 elections?

```python
train['Incumbent Party Vote Share'].mean()
```

What if we guessed this number for every election? How far off would this be on average?

```python
# Arrange y target vectors
target = 'Incumbent Party Vote Share'
y_train = train[target]
y_test = test[target]
```

```python
# Get mean baseline
print('Mean Baseline (using 0 features)')
guess = y_train.mean()
```

```python
# Train Error
from sklearn.metrics import mean_absolute_error
y_pred = [guess] * len(y_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Train Error (1952-2004 elections): {mae:.2f} percentage points')
```

```python
# Test Error
y_pred = [guess] * len(y_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test Error (2008-16 elections): {mae:.2f} percentage points')
```

### Simple regression (1 feature)

Make a scatterplot of the relationship between 1 feature and the target.

We'll use an economic feature: Average Recent Growth in Personal Incomes. ("Bread")

```python
import pandas as pd
import plotly.express as px

px.scatter(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004',
    trendline='ols',  # Ordinary Least Squares
)
```

1952 & 1968 are outliers: The incumbent party got fewer votes than predicted by the regression. What do you think could explain those years? We'll come back to this soon, but first...

Use scikit-learn to fit the simple regression with one feature.

Follow the [5 step process](https://jakevdp.github.io/PythonDataScienceHandbook/05.02-introducing-scikit-learn.html#Basics-of-the-API), and refer to [Scikit-Learn LinearRegression documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

```python
# 1. Import the appropriate estimator class from Scikit-Learn
from sklearn.linear_model import LinearRegression
```

```python
# 2. Instantiate this class
model = LinearRegression()
```

```python
# 3. Arrange X features matrices (already did y target vectors)
features = ['Average Recent Growth in Personal Incomes']
X_train = train[features]
X_test = test[features]
print(f'Linear Regression, dependent on: {features}')
```

```python
# 4. Fit the model
model.fit(X_train, y_train)
y_pred = model.predict(X_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Train Error: {mae:.2f} percentage points')
```

```python
# 5. Apply the model to new data
y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test Error: {mae:.2f} percentage points')
```

How does the error compare to the baseline?

### Multiple regression (2 features)

Make a scatterplot of the relationship between 2 features and the target.

We'll add another feature: US Military Fatalities per Million. ("Peace" or the lack thereof.)

Rotate the scatterplot to explore the data. What's different about 1952 & 1968?

```python
px.scatter_3d(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004'
)
```

Use scikit-learn to fit a multiple regression with two features.

```python
# TODO: Complete this cell

# Re-arrange X features matrices
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']
print(f'Linear Regression, dependent on: {features}')

```

```python
# TODO: Fit the model

```

```python
# TODO: Apply the model to new data

```

How does the error compare to the prior model?

### Plot the plane of best fit

For a regression with 1 feature, we plotted the line of best fit in 2D.

(There are many ways to do this. Plotly Express's `scatter` function makes it convenient with its `trendline='ols'` parameter.)

For a regression with 2 features, we can plot the plane of best fit in 3D!

(Plotly Express has a `scatter_3d` function but it won't plot the plane of best fit for us. But, we can write our own function, with the same "function signature" as the Plotly Express API.)

```python
import itertools
import numpy as np
import plotly.express as px
import plotly.graph_objs as go
from sklearn.linear_model import LinearRegression

def regression_3d(df, x, y, z, num=100, **kwargs):
    """
    Visualize linear regression in 3D: 2 features + 1 target

    df : Pandas DataFrame
    x : string, feature 1 column in df
    y : string, feature 2 column in df
    z : string, target column in df
    num : integer, number of quantiles for each feature
    """

    # Plot data
    fig = px.scatter_3d(df, x, y, z, **kwargs)

    # Fit Linear Regression
    features = [x, y]
    target = z
    model = LinearRegression()
    model.fit(df[features], df[target])

    # Define grid of coordinates in the feature space
    xmin, xmax = df[x].min(), df[x].max()
    ymin, ymax = df[y].min(), df[y].max()
    xcoords = np.linspace(xmin, xmax, num)
    ycoords = np.linspace(ymin, ymax, num)
    coords = list(itertools.product(xcoords, ycoords))

    # Make predictions for the grid
    predictions = model.predict(coords)
    Z = predictions.reshape(num, num).T

    # Plot predictions as a 3D surface (plane)
    fig.add_trace(go.Surface(x=xcoords, y=ycoords, z=Z))

    return fig
```

```python
regression_3d(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004'
)
```

Where are 1952 & 1968 in relation to the plane? Which elections are the biggest outliers now?

Roll over points on the plane to see predicted incumbent party vote share (z axis), dependent on personal income growth (x axis) and military fatatlies per capita (y axis).

### Get and interpret coefficients

During the previous module, we got the simple regression's coefficient and intercept. We plugged these numbers into an equation for the line of best fit, in slope-intercept form: $y = mx + b$

Let's review this objective, but now for multiple regression.

What's the equation for the plane of best fit?

$y = \beta_0 + \beta_1x_1 + \beta_2x_2$

Can you relate the intercept and coefficients to what you see in the plot above?

```python
model.intercept_, model.coef_
```

```python
beta0 = model.intercept_
beta1, beta2 = model.coef_
print(f'y = {beta0} + {beta1}x1 + {beta2}x2')
```

```python
# This is easier to read
print('Intercept', model.intercept_)
coefficients = pd.Series(model.coef_, features)
print(coefficients.to_string())
```

One of the coefficients is positive, and the other is negative. What does this mean?

Let's look at some scenarios. We'll see that one unit's change in an independent variable results in a coefficient worth of change in the dependent variable.

What does the model predict if income growth=0%, fatalities=0

```python
model.predict([[0, 0]])
```

Income growth = 1% (fatalities = 0)

```python
model.predict([[1, 0]])
```

The difference between these predictions = ?

```python
model.predict([[1, 0]]) - model.predict([[0, 0]])
```

What if... income growth = 2% (fatalities = 0)

```python
model.predict([[2, 0]])
```

The difference between these predictions = ?

```python
model.predict([[2, 0]]) - model.predict([[1, 0]])
```

What if... (income growth=2%) fatalities = 100

```python
model.predict([[2, 100]])
```

The difference between these predictions = ?

```python
model.predict([[2, 100]]) - model.predict([[2, 0]])
```

What if income growth = 3% (fatalities = 100)

```python
model.predict([[3, 100]])
```

The difference between these predictions = ?

```python
model.predict([[3, 100]]) - model.predict([[2, 100]])
```

What if (income growth = 3%) fatalities = 200

```python
model.predict([[3, 200]])
```

The difference between these predictions = ?

```python
model.predict([[3, 200]]) - model.predict([[3, 100]])
```

## Challenge

In your assignment, you'll fit a Linear Regression with at least 2 features.

# Understand how ordinary least squares regression minimizes the sum of squared errors

## Overview

So far, we've evaluated our models by their absolute error. It's an intuitive metric for regression problems.

However, ordinary least squares doesn't directly minimize absolute error. Instead, it minimizes squared error.

In this section, we'll introduce two new regression metrics:

- Squared error
- $R^2$

We'll demostrate two possible methods to minimize squared error:

- Guess & check
- Linear Algebra

## Follow Along

### Guess & Check

This function visualizes squared errors. We'll go back to simple regression with 1 feature, because it's much easier to visualize.

Use the function's m & b parameters to "fit the model" manually. Guess & check what values of m & b minimize squared error.

```python
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

def squared_errors(df, feature, target, m, b):
    """
    Visualize linear regression, with squared errors,
    in 2D: 1 feature + 1 target.

    Use the m & b parameters to "fit the model" manually.

    df : Pandas DataFrame
    feature : string, feature column in df
    target : string, target column in df
    m : numeric, slope for linear equation
    b : numeric, intercept for linear requation
    """

    # Plot data
    fig = plt.figure(figsize=(7,7))
    ax = plt.axes()
    df.plot.scatter(feature, target, ax=ax)

    # Make predictions
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    # Plot predictions
    ax.plot(x, y_pred)

    # Plot squared errors
    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)
    for x, y1, y2 in zip(x, y, y_pred):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))

    # Print regression metrics
    mse = mean_squared_error(y, y_pred)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Squared Error:', mse)
    print('Root Mean Squared Error:', rmse)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)
```

Here's what the mean baseline looks like:

```python
feature = 'Average Recent Growth in Personal Incomes'
squared_errors(train, feature, target, m=0, b=y_train.mean())
```

Notice that $R^2$ is exactly zero.

[$R^2$ represents the proportion of the variance for a dependent variable that is explained by the independent variable(s).](https://en.wikipedia.org/wiki/Coefficient_of_determination)

The mean baseline uses zero independent variables and explains none of the variance in the dependent variable, so its $R^2$ score is zero.

The highest possible $R^2$ score is 1. The lowest possible *Train* $R^2$ score with ordinary least squares regression is 0.

In this demo, it's possible to get a negative Train $R^2$, if you manually set values of m & b that are worse than the mean baseline. But that wouldn't happen in the real world.

However, in the real world, it _is_ possible to get a negative *Test/Validation* $R^2$. It means that your *Test/Validation* predictions are worse than if you'd constantly predicted the mean of the *Test/Validation* set.

---

Now that we've visualized the squared errors for the mean baseline, let's guess & check some better values for the m & b parameters:

```python
squared_errors(train, feature, target, m=3, b=46)
```

You can run the function repeatedly, with different values for m & b.

How do you interpret each metric you see?

- Mean Squared Error
- Root Mean Squared Error
- Mean Absolute Error
- $R^2$

Does guess & check really get used in machine learning? Sometimes! Some complex functions are hard to minimize, so we use a sophisticated form of guess & check called "gradient descent", which you'll learn about in Unit 4.

Fortunately, we don't need to use guess & check for ordinary least squares regression. We have a solution, using linear algebra!

### Linear Algebra

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

\begin{align}
\hat{\beta} = (X^{T}X)^{-1}X^{T}y
\end{align}

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

#### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

#### Lets calculate our $\beta$ parameters with numpy!

```python
# This is NOT something you'll be tested on. It's just a demo.

# X is a matrix. Add column of constants for fitting the intercept.
def add_constant(X):
    constant = np.ones(shape=(len(X),1))
    return np.hstack((constant, X))
X = add_constant(train[features].values)
print('X')
print(X)

# y is a column vector
y = train[target].values[:, np.newaxis]
print('y')
print(y)

# Least squares solution in code
X_transpose = X.T
X_transpose_X = X_transpose @ X
X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
X_transpose_y = X_transpose @ y
beta_hat = X_transpose_X_inverse @ X_transpose_y

print('Beta Hat')
print(beta_hat)
```

```python
# Scikit-learn gave the exact same results!
model.intercept_, model.coef_
```

# Define overfitting/underfitting and the bias/variance tradeoff

## Overview

Read [_Python Data Science Handbook,_ Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#The-Bias-variance-trade-off). Jake VanderPlas explains overfitting & underfitting:

> Fundamentally, the question of "the best model" is about finding a sweet spot in the tradeoff between bias and variance. Consider the following figure, which presents two regression fits to the same dataset:
>
>
>
> The model on the left attempts to find a straight-line fit through the data. Because the data are intrinsically more complicated than a straight line, the straight-line model will never be able to describe this dataset well. Such a model is said to _underfit_ the data: that is, it does not have enough model flexibility to suitably account for all the features in the data; another way of saying this is that the model has high _bias_.
>
> The model on the right attempts to fit a high-order polynomial through the data. Here the model fit has enough flexibility to nearly perfectly account for the fine features in the data, but even though it very accurately describes the training data, its precise form seems to be more reflective of the particular noise properties of the data rather than the intrinsic properties of whatever process generated that data. Such a model is said to _overfit_ the data: that is, it has so much model flexibility that the model ends up accounting for random errors as well as the underlying data distribution; another way of saying this is that the model has high _variance_.

VanderPlas goes on to connect these concepts to the "bias/variance tradeoff":

> From the scores associated with these two models, we can make an observation that holds more generally:
>
>- For high-bias models, the performance of the model on the validation set is similar to the performance on the training set.
>
>- For high-variance models, the performance of the model on the validation set is far worse than the performance on the training set.
>
> If we imagine that we have some ability to tune the model complexity, we would expect the training score and validation score to behave as illustrated in the following figure:
>
>
>
> The diagram shown here is often called a validation curve, and we see the following essential features:
>
>- The training score is everywhere higher than the validation score. This is generally the case: the model will be a better fit to data it has seen than to data it has not seen.
>- For very low model complexity (a high-bias model), the training data is under-fit, which means that the model is a poor predictor both for the training data and for any previously unseen data.
>- For very high model complexity (a high-variance model), the training data is over-fit, which means that the model predicts the training data very well, but fails for any previously unseen data.
>- For some intermediate value, the validation curve has a maximum. This level of complexity indicates a suitable trade-off between bias and variance.
>
>The means of tuning the model complexity varies from model to model.

So far, our only "means of tuning the model complexity" has been selecting one feature or two features for our linear regression models. But we'll quickly start to select more features, and more complex models, with more "hyperparameters."

This is just a first introduction to underfitting & overfitting. We'll continue to learn about this topic all throughout this unit.

## Follow Along

Let's make our own Validation Curve, by tuning a new type of model complexity: polynomial degrees in a linear regression.

Go back to the the NYC Tribeca condo sales data

```python
# Read NYC Tribeca condo sales data, from first 4 months of 2019.
# Dataset has 90 rows, 9 columns.
df = pd.read_csv(DATA_PATH+'condos/tribeca.csv')
assert df.shape == (90, 9)

# Arrange X features matrix & y target vector
features = ['GROSS_SQUARE_FEET']
target = 'SALE_PRICE'
X = df[features]
y = df[target]
```

Do random [train/test split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=11)
```

Repeatedly fit increasingly complex models, and keep track of the scores

```python
from IPython.display import display, HTML
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

# Credit for PolynomialRegression: Jake VanderPlas, Python Data Science Handbook, Chapter 5.3
# https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#Validation-curves-in-Scikit-Learn
def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree),
                         LinearRegression(**kwargs))

polynomial_degrees = range(1, 10, 2)
train_r2s = []
test_r2s = []

for degree in polynomial_degrees:
    model = PolynomialRegression(degree)
    display(HTML(f'Polynomial degree={degree}'))

    model.fit(X_train, y_train)
    train_r2 = model.score(X_train, y_train)
    test_r2 = model.score(X_test, y_test)
    display(HTML(f'<b style="color: blue">Train R2 {train_r2:.2f}</b>'))
    display(HTML(f'<b style="color: red">Test R2 {test_r2:.2f}</b>'))

    plt.scatter(X_train, y_train, color='blue', alpha=0.5)
    plt.scatter(X_test, y_test, color='red', alpha=0.5)
    plt.xlabel(features)
    plt.ylabel(target)

    x_domain = np.linspace(X.min(), X.max())
    curve = model.predict(x_domain)
    plt.plot(x_domain, curve, color='blue')
    plt.show()
    display(HTML('<hr/>'))

    train_r2s.append(train_r2)
    test_r2s.append(test_r2)

display(HTML('Validation Curve'))
plt.plot(polynomial_degrees, train_r2s, color='blue', label='Train')
plt.plot(polynomial_degrees, test_r2s, color='red', label='Test')
plt.xlabel('Model Complexity (Polynomial Degree)')
plt.ylabel('R^2 Score')
plt.legend()
plt.show()
```

As model complexity increases, what happens to Train $R^2$ and Test $R^2$?

# Review

In your assignment, you'll continue to **predict how much it costs to rent an apartment in NYC,** using the dataset from renthop.com.

- Do train/test split. Use data from April & May 2016 to train. Use data from June 2016 to test.
- Engineer at least two new features.
- Fit a linear regression model with at least two features.
- Get the model's coefficients and intercept.
- Get regression metrics RMSE, MAE, and $R^2$, for both the train and test sets.

You've been provided with a separate notebook for your assignment, which has all the instructions and stretch goals. What's the best test MAE you can get? Share your score and features used with your cohort on Slack!

# Sources

#### Train/Test Split
- James, Witten, Hastie, Tibshirani, [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/), Chapter 2.2, Assessing Model Accuracy
- Hyndman, Athanasopoulos, [_Forecasting,_ Chapter 3.4,](https://otexts.com/fpp2/accuracy.html) Evaluating forecast accuracy
- Rachel Thomas, [How (and why) to create a good validation set](https://www.fast.ai/2017/11/13/validation-sets/)

#### Bias-Variance Tradeoff
- Jake VanderPlas, [_Python Data Science Handbook,_ Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#The-Bias-variance-trade-off), Hyperparameters and Model Validation
- StatQuest, [Machine Learning Fundamentals: Bias and Variance](https://youtu.be/EuBBz3bI-aA) (6.5 minutes)

#### "Bread and Peace" Background
- Douglas Hibbs, [Background Information on the ‘Bread and Peace’ Model of Voting in Postwar US Presidential Elections](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)
- Nate Silver, [What Do Economic Models Really Tell Us About Elections?](https://fivethirtyeight.com/features/what-do-economic-models-really-tell-us-about-elections/)

#### "Bread and Peace" Data Sources & Definitions
- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

> Fatalities denotes the cumulative number of American military fatalities per millions of US population the in Korea, Vietnam, Iraq and Afghanistan wars during the presidential terms preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and 2012 elections. —[Hibbs](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 33