# Lab 01 - Local Optimization
## Tasks
- Introduction to PyTorch - will be used throughout the course
- Introduction to autograd
- Implement gradient descent using autograd framework
- Optimize a simple quadrupole triplet

## Set up environment

```python
!pip install git+https://github.com/uspas/2021_optimization_and_ml --quiet
```

```python
%reset -f

import numpy as np
import matplotlib.pyplot as plt

#matplotlib graphs will be included in your notebook, next to the code:
%matplotlib inline

#import toy accelerator package
from uspas_ml.accelerator_toy_models import simple_lattices

import torch
```

## Introduction to PyTorch
PyTorch is a machine learning library which implements autograd functionality (more on this in a bit). It will be useful for creating and training surrogate models. Many function implemented in numpy for `ndarrays` are also implemented in PyTorch with the `torch.Tensor` class. Here we show some examples of using `torch.Tensor`.

```python
data = [[1, 2],[3, 4]]
x_data = torch.tensor(data)
```

```python
np_array = np.array(data)
x_np = torch.from_numpy(np_array)
```

```python
x_ones = torch.ones_like(x_data) # retains the properties of x_data
print(f"Ones Tensor: \n {x_ones} \n")

x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data
print(f"Random Tensor: \n {x_rand} \n")
```

```python
shape = (2,3,)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")
```

```python
tensor = torch.rand(3,4)

print(f"Shape of tensor: {tensor.shape}")
print(f"Datatype of tensor: {tensor.dtype}")
print(f"Device tensor is stored on: {tensor.device}")
```

```python
tensor = torch.ones(4, 4)
print('First row: ',tensor[0])
print('First column: ', tensor[:, 0])
print('Last column:', tensor[..., -1])
tensor[:,1] = 0
print(tensor)
```

```python
t1 = torch.cat([tensor, tensor, tensor], dim=1)
print(t1)
```

```python
# This computes the matrix multiplication between two tensors. y1, y2, y3 will have the same value
y1 = tensor @ tensor.T
y2 = tensor.matmul(tensor.T)

y3 = torch.rand_like(tensor)
torch.matmul(tensor, tensor.T, out=y3)

# This computes the element-wise product. z1, z2, z3 will have the same value
z1 = tensor * tensor
z2 = tensor.mul(tensor)

z3 = torch.rand_like(tensor)
torch.mul(tensor, tensor, out=z3)
```

```python
# This sums all the elements of a tensor
agg = tensor.sum()
agg_item = agg.item()
print(agg_item, type(agg_item))
```

```python
# Add a scalar to all the elements of a tensor
print(tensor, "\n")
tensor.add_(5)
print(tensor)
tensor += 5
print(tensor)
```

## A note regarding tensor shapes
Paying attention to the shape of your tensors/numpy arrays is important, especially if you don't have a lot of experience programming. In general ML algorithms accept 2D arrays where the shape is given by `(number_of_samples, dimension_of_samples)`. Even if the input parameter is 1D the input shape should be `(number_of_samples, 1)`.

Furthermore it is important to remember that slicing along a single axis returns a 1D tensor/array ie. `x[:,0]` returns an array shape of `(n_samples,)`.

Functions that will be useful for manipulating the shape of a tensor/array:
- `.reshape()`
- `.unsqueeze()`
- `.squeeze()`
- `.flatten()`

## GPU Integration
To speed up many of the computations that we will do in this course we will move our data/models to a GPU. This works best when doing large matrix calculations or batch optimization routines. Keep in mind the time cost associated with moving data back and forth between a CPU and a GPU, this can slow down smaller scale calculations.

```python
# We move our tensor to the GPU if available
if torch.cuda.is_available():
    tensor = tensor.to('cuda')
```

## Autograd with PyTorch
Its hard to overstate the importance of Automatic Differentiation (commonly referred to Autograd) to the machine learning community. It forms the basis for training/optimizing surrogate models such as Gaussian Processes and Neural Networks.

Autograd is a framework for doing calculations while keeping track of the derivatives associated with each calculation step. We will then be able to calculate the total derivative of say, the beam size vs. a quadrupole strength parameter. This will be used to train surrogate models with derivative based optimization methods (gradient descent and its descendants).

Autograd works by creating what is known by a computational graph (an example of which is below). The graph has a set of what are known as leaf nodes (or tensors) $a,w_1,w_2,w_3,w_4$ which represent input variables that we choose. We wish to compute how changing these variables will effect the result $L$. Each of the lines on the computation graph represent a single calculation which we will need the derivate of to calculate the total derivative.

Our calculation will involve a number of discrete steps, each which create intermediate variables seen below:

$
\begin{align}
    b & = w_1 * a\\
    c & = w_2 * a\\
    d & = w_3*b + w_4*c\\
    L & = 10 -d
\end{align}
$

In order to calculate the total derivative of L with respect to say $w_1$ we use the chain rule

$$
\frac{dL}{dw_1} = \frac{\partial L}{\partial d}\frac{\partial d}{\partial b}\frac{\partial b}{\partial w_1}
$$

Using PyTorch the partial derivatives for each calculation are easily calculated for a wide variety of calculations and variable manipulations that we need for model building and programming. Furthermore, this is easily done with thousands of variables and/or manipulations at a time! Thanks PyTorch!

Lets take a look at how this can be done in practice. We start with defining some variables that we will need the derivatives with respect to by specifying `requires_grad = True` during creation.

```python
#define variables (the chosen values here are arbitrary)
a = torch.tensor([2.], requires_grad=True)
w1 = torch.tensor([3.], requires_grad=True)
w2 = torch.tensor([1.], requires_grad=True)
w3 = torch.tensor([0.5], requires_grad=True)
w4 = torch.tensor([-2.], requires_grad=True)

#do the calculation
b = w1*a
c = w2*a
d = w3*b + w4*c
L = 10 - d
```

To calculate the derivatives we call `L.backward()`. After calling `backward()` the gradients are stored in the `.grad` attribute of each variable.

```python
L.backward()
print(a.grad)
print(w4.grad)
```

Note that we are calculating the gradient of $a$ and $w_4$ evaluated at their currently set values. In other words the grad attribute gives us

$$
\frac{dL}{da}\Big\rvert_{a = 2}
$$

**Task:**
    Calculate (by hand) the quantity $\frac{dL}{dw_1}$, and check that this is consistent with the result given by `w1.grad`.

```python
print(w1.grad) # Check that this is consistent with your analytical calculation
```

Note that, if you redo the calculation a second time, the gradient will **accumulate**:

```python
# redo the calculation a second time
b = w1*a
c = w2*a
d = w3*b + w4*c
L = 10 - d
# call backward to calculate the gradients a second time
L.backward()
# print gradient
print(w1.grad)
```

If you want to avoid this, you can call `.grad.zero_()`

```python
# zero the gradient
w1.grad.zero_()
# redo the calculation a third time
b = w1*a
c = w2*a
d = w3*b + w4*c
L = 10 - d
# call backward to calculate the gradients a third time
L.backward()
# print gradient
print(w1.grad)
```

Note that you can also ask `pytorch` to temporarily stop tracking the gradients for certain operations:

```python
with torch.no_grad():
    w1 = 2*w1
```

## Gradient descent

**Task:**
    Now its your turn! Implement a gradient descent algortihm to minimize a simple test function below using autograd. Complete the implementation of the function `gradient_descent` below, to perform 100 iterations of gradient descent.

- implement the gradient descent algorithm using the above-mentioned features of pytorch
- record the values of `X0` as you go and generate a plot that shows how gradient descent travels through input space
- given the form of `test_function`, which value do you expect `X0` to converge to?
- redo the plot for 3 different values of the step size `alpha` [0.005, 0.1, 1.0 ]

```python
def test_function(X):
    """
    Function to be optimized.

    Input: X, a pytorch tensor of shape (2,)
    Output: a single scalar
    """
    return (X[0] - 0.5)**2 + X[1]**2

def gradient_descent(X0, function, alpha=0.1, n_iterations=100):
    """
    Performs 100 iterations of gradient descent

    Inputs:
    X0: a pytorch tensor to be modified in-place by the gradient descent algorithm
    function: a callable, that takes X0 as input
    alpha: step size for the gradient descent algorithm
    n_iterations: number of iterations of gradient descent

    Returns:
    history_X: array of values of X
    history_f: array of values of f
    """
    history_X = []
    history_f = []
    for iteration in range(n_iterations): # iterations of gradient descent
        f = function(X0)

        # Your code that modifies X0 in-place here
        # ...

        # Keep a copy in history (do not modify these lines)
        history_X.append( X0.detach().numpy().copy() )
        history_f.append( f.detach().numpy().copy() )

    return np.array(history_X), np.array(history_f)
```

```python
# Test your code by executing the code below
X0 = torch.tensor([0.9,0.9], requires_grad=True)
alpha = 0.1

history_X, history_f = gradient_descent( X0, test_function, alpha)

# Plot the trajectory of the points
plt.scatter( history_X[:,0], history_X[:,1], c=np.arange(len(history_X)))
plt.xlim(-1, 1)
plt.ylim(-1, 1)
```

## Accelerator example
Now we will use gradient descent to optimize the strengths of quadrupoles in a triplet configuration.

The function below creates a quadrupole triplet as defined in simple_lattices.py (see the accelerator toy models) and returns $\sqrt{\sigma_x^2 + \sigma_y^2}$ for an initial beam matrix defined in the function below.

Remember that the beam matrix takes the form of

$
\Sigma =
\begin{bmatrix}
\sigma_{11} & \sigma_{12} & \sigma_{13} & \sigma_{14} & \sigma_{15} & \sigma_{16} \\
\sigma_{21} & \sigma_{22} & \sigma_{23} & \sigma_{24} & \sigma_{25} & \sigma_{26} \\
\sigma_{31} & \sigma_{32} & \sigma_{33} & \sigma_{34} & \sigma_{35} & \sigma_{36} \\
\sigma_{41} & \sigma_{42} & \sigma_{43} & \sigma_{44} & \sigma_{45} & \sigma_{46} \\
\sigma_{51} & \sigma_{52} & \sigma_{53} & \sigma_{54} & \sigma_{55} & \sigma_{56} \\
\sigma_{61} & \sigma_{62} & \sigma_{63} & \sigma_{64} & \sigma_{65} & \sigma_{66} \\
\end{bmatrix}
$

where for example $\sigma_{11} = \sigma_x^2, \sigma_{12} = \sigma_{x,x'}$ etc.

We start with the following initial beam matrix.
$
\Sigma =
\begin{bmatrix}
1e-6 & 0 & 0 & 0 & 0 & 0 \\
0 & 1e-8 & 0 & 0 & 0 & 0 \\
0 & 0 & 1e-6 & 0 & 0 & 0 \\
0 & 0 & 0 & 1e-8 & 0 & 0 \\
0 & 0 & 0 & 0 & 1e-8 & 0 \\
0 & 0 & 0 & 0 & 0 & 1e-8 \\
\end{bmatrix}
$

```python
def beamsize(K , noise = 0.0):
    '''
    calculate sqrt(sigma_x^2 + sigma_y^2)

    K : magnetic strength of each quad magnet, torch.tensor, shape (3,)
    noise : rms noise level of measurement, float (default 0.0)

    Returns: the beam size in mm
    '''

    #generate initial beam matrix
    # - x/y/z geometric emittance of 1.0e-8 m-rad
    init_beam_matrix = torch.eye(6) * 1.0e-8

    #set x_rms beam size to 1 mm and rms divergence to 0.1 mrad
    init_beam_matrix[0,0] = 1.0e-3 ** 2
    init_beam_matrix[1,1] = 1.0e-4 ** 2
    init_beam_matrix[2,2] = 1.0e-3 ** 2
    init_beam_matrix[3,3] = 1.0e-4 ** 2

    #create accelerator lattice object with one quad and a drift
    line = simple_lattices.create_triplet(K)

    #propagate beam matrix
    final_beam_matrix = line.propagate_beam_matrix(init_beam_matrix, noise)
    return 1.e3*torch.sqrt(final_beam_matrix[0,0] + final_beam_matrix[2,2]) #Convert to mm
```

```python
#example usage
K = torch.tensor([1.0, -1.0, 0.5], requires_grad = True)
print(K.requires_grad)
size = beamsize(K)
print(size)
size.backward()
print(K.grad)
```

## Gradient Descent on the Accelerator Example

**Task:**
    Now apply gradient descent to optimizing the quadrupole strengths to minimize the total final beam size using the above function. Write the corresponding code below. (Feel free to reuse the function `gradient_descent` and to tune `alpha` and `n_iterations` to reach convergence - which you can assess by plotting the returned history ; do not hesitate to use a large value for `alpha`.)

What is the value of the beamsize that is obtained at the end of the optimization?

```python
K = torch.tensor([1.0, -1.0, 0.5], requires_grad = True)
# Your code here
# ...
```

**Your answer here:** (What is the value of the beamsize that is obtained at the end of the optimization?)

## Gradient descent with numerical differentiation

**Homework:**
    Try to minimize again the beam size, but using numerical differentiation instead of autograd. In this case, you will **not** call the `.backward` and `.grad` functions, but instead you will need to calculate the gradient numerically: vary in the input `K` by `h=1.e-4` in each direction, in order to compute the gradient of `beam_size`.

(Note that, because we do not call `.backward`, we also do not need to set `requires_grad = True`.)

Does the algorithm reach the same final value, for the beamsize?

```python
def gradient_descent_nd(X0, function, alpha=0.1, n_iterations=100):
    """
    Performs n_iterations iterations of gradient descent,
    with numerical differentiation

    Inputs:
    X0: a pytorch tensor to be modified in-place by the gradient descent algorithm
    function: a callable, that takes X0 as input
    alpha: step size for the gradient descent algorithm
    n_iterations: number of iterations of gradient descent

    Returns:
    history_X: array of values of X
    history_f: array of values of f
    """
    history_X = []
    history_f = []
    for iteration in range(n_iterations): # iterations of gradient descent
        f = function(X0)

        # Your code that modifies X0 in-place here
        # ...

        # Keep a copy in history (do not modify these lines)
        history_X.append( X0.detach().numpy().copy() )
        history_f.append( f.detach().numpy().copy() )

    return np.array(history_X), np.array(history_f)
```

```python
# Your code here: call `gradient_descent_nd` and plot the corresponding `history_f`
```

**Your answer here:** (Does the algorithm reach the same value of `K`?)

## Gradient descent with noise

The function below emulates noise in the beam size measurement.
Call this function several times for a given value of `K`.

```python
def beamsize_with_noise(K):
    return beamsize(K, noise=2.e-5)
```

```python
# Call this function several times to get a sense of how much
# the noise is changing the result
beamsize_with_noise(K)
```

**Homework:**
    Now perform the same optimization as before, but on the function `beam_size_with_noise` instead of `beam_size`. Does the analytical gradient descent reach approximately the same final value of the beamsize as before? Does the version of gradient descent with finite difference reach the same value too? Why? (Plot the history of the beamsize in both cases.)

```python
# Your code here
# ...
```

**Your answer here:** (Does the analytical gradient descent reach approximately the same final value of the beamsize as before? Does the version of gradient descent with finite difference reach the same value too? Why?)


# Homework and bake-off: word-level entailment with neural networks

```python
__author__ = "Christopher Potts"
__version__ = "CS224u, Stanford, Fall 2020"
```

## Contents

1. [Overview](#Overview)
1. [Set-up](#Set-up)
1. [Data](#Data)
1. [Baseline](#Baseline)
  1. [Representing words: vector_func](#Representing-words:-vector_func)
  1. [Combining words into inputs: vector_combo_func](#Combining-words-into-inputs:-vector_combo_func)
  1. [Classifier model](#Classifier-model)
  1. [Baseline results](#Baseline-results)
1. [Homework questions](#Homework-questions)
  1. [Hypothesis-only baseline [2 points]](#Hypothesis-only-baseline-[2-points])
  1. [Alternatives to concatenation [2 points]](#Alternatives-to-concatenation-[2-points])
  1. [A deeper network [2 points]](#A-deeper-network-[2-points])
  1. [Your original system [3 points]](#Your-original-system-[3-points])
1. [Bake-off [1 point]](#Bake-off-[1-point])

## Overview

The general problem is word-level natural language inference. Training examples are pairs of words $(w_{L}, w_{R}), y$ with $y = 1$ if $w_{L}$ entails $w_{R}$, otherwise $0$.

The homework questions below ask you to define baseline models for this and develop your own system for entry in the bake-off, which will take place on a held-out test-set distributed at the start of the bake-off. (Thus, all the data you have available for development is available for training your final system before the bake-off begins.)

## Set-up

See [the first notebook in this unit](nli_01_task_and_data.ipynb) for set-up instructions.

```python
from collections import defaultdict
import json
import numpy as np
import os
import pandas as pd
from torch_shallow_neural_classifier import TorchShallowNeuralClassifier
import nli
import utils
```

```python
DATA_HOME = 'data'

NLIDATA_HOME = os.path.join(DATA_HOME, 'nlidata')

wordentail_filename = os.path.join(
    NLIDATA_HOME, 'nli_wordentail_bakeoff_data.json')

GLOVE_HOME = os.path.join(DATA_HOME, 'glove.6B')
```

## Data

I've processed the data into a train/dev split that is designed to put some pressure on our models to actually learn these semantic relations, as opposed to exploiting regularities in the sample.

The defining feature of the dataset is that the `train` and `dev` __vocabularies__ are disjoint. That is, if a word `w` appears in a training pair, it does not occur in any text pair. It follows from this that there are also no word-pairs shared between train and dev, as you would expect. This should require your models to learn abstract relationships, as opposed to memorizing incidental properties of individual words in the dataset.

```python
with open(wordentail_filename) as f:
    wordentail_data = json.load(f)
```

The keys are the splits plus a list giving the vocabulary for the entire dataset:

```python
wordentail_data.keys()
```

    dict_keys(['dev', 'train', 'vocab'])

```python
wordentail_data['train'][: 5]
```

    [[['abode', 'house'], 1],
     [['abortion', 'anaemia'], 0],
     [['abortion', 'aneurysm'], 0],
     [['abortion', 'blindness'], 0],
     [['abortion', 'deafness'], 0]]

```python
nli.get_vocab_overlap_size(wordentail_data)
```

    0

Because no words are shared between `train` and `dev`, no pairs are either:

```python
nli.get_pair_overlap_size(wordentail_data)
```

    0

Here is the label distribution:

```python
pd.DataFrame(wordentail_data['train'])[1].value_counts()
```

    0    7000
    1    1283
    Name: 1, dtype: int64

This is a challenging label distribution – there are more than 5 times as more non-entailment cases as entailment cases.

## Baseline

Even in deep learning, __feature representation is vital and requires care!__ For our task, feature representation has two parts: representing the individual words and combining those representations into a single network input.

### Representing words: vector_func

Let's consider two baseline word representations methods:

1. Random vectors (as returned by `utils.randvec`).
1. 50-dimensional GloVe representations.

```python
def randvec(w, n=50, lower=-1.0, upper=1.0):
    """Returns a random vector of length `n`. `w` is ignored."""
    return utils.randvec(n=n, lower=lower, upper=upper)
```

```python
def load_glove50():
    glove_src = os.path.join(GLOVE_HOME, 'glove.6B.50d.txt')
    # Creates a dict mapping strings (words) to GloVe vectors:
    GLOVE = utils.glove2dict(glove_src)
    return GLOVE

GLOVE = load_glove50()

def glove_vec(w):
    """Return `w`'s GloVe representation if available, else return
    a random vector."""
    return GLOVE.get(w, randvec(w, n=50))
```

### Combining words into inputs: vector_combo_func

Here we decide how to combine the two word vectors into a single representation. In more detail, where `u` is a vector representation of the left word and `v` is a vector representation of the right word, we need a function `vector_combo_func` such that `vector_combo_func(u, v)` returns a new input vector `z` of dimension `m`. A simple example is concatenation:

```python
def vec_concatenate(u, v):
    """Concatenate np.array instances `u` and `v` into a new np.array"""
    return np.concatenate((u, v))
```

`vector_combo_func` could instead be vector average, vector difference, etc. (even combinations of those) – there's lots of space for experimentation here; [homework question 2](#Alternatives-to-concatenation-[2-points]) below pushes you to do some exploration.

### Classifier model

For a baseline model, I chose `TorchShallowNeuralClassifier`:

```python
net = TorchShallowNeuralClassifier(early_stopping=True)
```

### Baseline results

The following puts the above pieces together, using `vector_func=glove_vec`, since `vector_func=randvec` seems so hopelessly misguided for our problem!

```python
baseline_experiment = nli.wordentail_experiment(
    train_data=wordentail_data['train'],
    assess_data=wordentail_data['dev'],
    model=net,
    vector_func=glove_vec,
    vector_combo_func=vec_concatenate)
```

    Stopping after epoch 140. Validation score did not improve by tol=1e-05 for more than 10 epochs. Final error is 1.023185983300209

                  precision    recall  f1-score   support

               0      0.870     0.950     0.908      1732
               1      0.506     0.263     0.346       334

        accuracy                          0.839      2066
       macro avg      0.688     0.607     0.627      2066
    weighted avg      0.811     0.839     0.818      2066

## Homework questions

Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)

### Hypothesis-only baseline [2 points]

During our discussion of SNLI and MultiNLI, we noted that a number of research teams have shown that hypothesis-only baselines for NLI tasks can be remarkably robust. This question asks you to explore briefly how this baseline affects our task.

For this problem, submit two functions:

1. A `vector_combo_func` function called `hypothesis_only` that simply throws away the premise, using the unmodified hypothesis (second) vector as its representation of the example.

1. A function called `run_hypothesis_only_evaluation` that does the following:
    1. Loops over the two `vector_combo_func` values `vec_concatenate` and `hypothesis_only`, calling `nli.wordentail_experiment` to train on the 'train' portion and assess on the 'dev' portion, with `glove_vec` as the `vector_func`. So that the results are consistent, use an `sklearn.linear_model.LogisticRegression` with default parameters as the model.
    1. Returns a `dict` mapping `function_name` strings to the 'macro-F1' score for that pair, as returned by the call to `nli.wordentail_experiment`. (Tip: you can get the `str` name of, e.g., `hypothesis_only` with `hypothesis_only.__name__`.)

The functions `test_hypothesis_only` and `test_run_hypothesis_only_evaluation` will help ensure that your functions have the desired logic.

```python
def hypothesis_only(u, v):
    ##### YOUR CODE HERE
    return v

def run_hypothesis_only_evaluation():
    ##### YOUR CODE HERE
    from sklearn.linear_model import LogisticRegression
    result = dict()
    lr = LogisticRegression()
    for combo in [hypothesis_only, vec_concatenate]:
        experiment = nli.wordentail_experiment(
            train_data=wordentail_data['train'],
            assess_data=wordentail_data['dev'],
            model=lr,
            vector_func=glove_vec,
            vector_combo_func=combo)
        result[combo.__name__] = experiment['macro-F1']
    return result

```

```python
def test_hypothesis_only(hypothesis_only):
    v = hypothesis_only(1, 2)
    assert v == 2
```

```python
test_hypothesis_only(hypothesis_only)
```

```python
def test_run_hypothesis_only_evaluation(run_hypothesis_only_evaluation):
    results = run_hypothesis_only_evaluation()
    assert all(x in results for x in ('hypothesis_only', 'vec_concatenate')), \
        ("The return value of `run_hypothesis_only_evaluation` does not "
         "have the intended kind of keys.")
    assert isinstance(results['vec_concatenate'], float), \
        ("The values of the `run_hypothesis_only_evaluation` result "
         "should be floats.")
```

```python
test_run_hypothesis_only_evaluation(run_hypothesis_only_evaluation)
```

                  precision    recall  f1-score   support

               0      0.853     0.971     0.908      1732
               1      0.462     0.129     0.201       334

        accuracy                          0.835      2066
       macro avg      0.657     0.550     0.555      2066
    weighted avg      0.789     0.835     0.794      2066

                  precision    recall  f1-score   support

               0      0.863     0.954     0.906      1732
               1      0.473     0.213     0.293       334

        accuracy                          0.834      2066
       macro avg      0.668     0.583     0.600      2066
    weighted avg      0.800     0.834     0.807      2066

### Alternatives to concatenation [2 points]

We've so far just used vector concatenation to represent the premise and hypothesis words. This question asks you to explore two simple alternative:

1. Write a function `vec_diff` that, for a given pair of vector inputs `u` and `v`, returns the element-wise difference between `u` and `v`.

1. Write a function `vec_max` that, for a given pair of vector inputs `u` and `v`, returns the element-wise max values between `u` and `v`.

You needn't include your uses of `nli.wordentail_experiment` with these functions, but we assume you'll be curious to see how they do!

```python
def vec_diff(u, v):
    ##### YOUR CODE HERE
    return u - v

def vec_max(u, v):
    ##### YOUR CODE HERE
    return np.maximum(u, v)

```

```python
def test_vec_diff(vec_diff):
    u = np.array([10.2, 8.1])
    v = np.array([1.2, -7.1])
    result = vec_diff(u, v)
    expected = np.array([9.0, 15.2])
    assert np.array_equal(result, expected), \
        "Expected {}; got {}".format(expected, result)
```

```python
test_vec_diff(vec_diff)
```

```python
def test_vec_max(vec_max):
    u = np.array([1.2,  8.1])
    v = np.array([10.2, -7.1])
    result = vec_max(u, v)
    expected = np.array([10.2, 8.1])
    assert np.array_equal(result, expected), \
        "Expected {}; got {}".format(expected, result)
```

```python
test_vec_max(vec_max)
```

### A deeper network [2 points]

It is very easy to subclass `TorchShallowNeuralClassifier` if all you want to do is change the network graph: all you have to do is write a new `build_graph`. If your graph has new arguments that the user might want to set, then you should also redefine `__init__` so that these values are accepted and set as attributes.

For this question, please subclass `TorchShallowNeuralClassifier` so that it defines the following graph:

$$\begin{align}
h_{1} &= xW_{1} + b_{1} \\
r_{1} &= \textbf{Bernoulli}(1 - \textbf{dropout_prob}, n) \\
d_{1} &= r_1 * h_{1} \\
h_{2} &= f(d_{1}) \\
h_{3} &= h_{2}W_{2} + b_{2}
\end{align}$$

Here, $r_{1}$ and $d_{1}$ define a dropout layer: $r_{1}$ is a random binary vector of dimension $n$, where the probability of a value being $1$ is given by $1 - \textbf{dropout_prob}$. $r_{1}$ is multiplied element-wise by our first hidden representation, thereby zeroing out some of the values. The result is fed to the user's activation function $f$, and the result of that is fed through another linear layer to produce $h_{3}$. (Inside `TorchShallowNeuralClassifier`, $h_{3}$ is the basis for a softmax classifier; no activation function is applied to it because the softmax scaling is handled internally by the loss function.)

For your implementation, please use `nn.Sequential`, `nn.Linear`, and `nn.Dropout` to define the required layers.

For comparison, using this notation, `TorchShallowNeuralClassifier` defines the following graph:

$$\begin{align}
h_{1} &= xW_{1} + b_{1} \\
h_{2} &= f(h_{1}) \\
h_{3} &= h_{2}W_{2} + b_{2}
\end{align}$$

The following code starts this sub-class for you, so that you can concentrate on `build_graph`. Be sure to make use of `self.dropout_prob`.

For this problem, submit just your completed  `TorchDeepNeuralClassifier`. You needn't evaluate it, though we assume you will be keen to do that!

You can use `test_TorchDeepNeuralClassifier` to ensure that your network has the intended structure.

```python
import torch.nn as nn

class TorchDeepNeuralClassifier(TorchShallowNeuralClassifier):
    def __init__(self, dropout_prob=0.7, **kwargs):
        self.dropout_prob = dropout_prob
        super().__init__(**kwargs)

    def build_graph(self):
        """Complete this method!

        Returns
        -------
        an `nn.Module` instance, which can be a free-standing class you
        write yourself, as in `torch_rnn_classifier`, or the outpiut of
        `nn.Sequential`, as in `torch_shallow_neural_classifier`.

        """
        ##### YOUR CODE HERE
        return nn.Sequential(
            nn.Linear(self.input_dim, self.hidden_dim),
            nn.Dropout(p=self.dropout_prob),
            self.hidden_activation,
            nn.Linear(self.hidden_dim, self.n_classes_))

```

```python
def test_TorchDeepNeuralClassifier(TorchDeepNeuralClassifier):
    dropout_prob = 0.55
    assert hasattr(TorchDeepNeuralClassifier(), "dropout_prob"), \
        "TorchDeepNeuralClassifier must have an attribute `dropout_prob`."
    try:
        inst = TorchDeepNeuralClassifier(dropout_prob=dropout_prob)
    except TypeError:
        raise TypeError("TorchDeepNeuralClassifier must allow the user "
                        "to set `dropout_prob` on initialization")
    inst.input_dim = 10
    inst.n_classes_ = 5
    graph = inst.build_graph()
    assert len(graph) == 4, \
        "The graph should have 4 layers; yours has {}".format(len(graph))
    expected = {
        0: 'Linear',
        1: 'Dropout',
        2: 'Tanh',
        3: 'Linear'}
    for i, label in expected.items():
        name = graph[i].__class__.__name__
        assert label in name, \
            ("The {} layer of the graph should be a {} layer; "
            "yours is {}".format(i, label, name))
    assert graph[1].p == dropout_prob, \
        ("The user's value for `dropout_prob` should be the value of "
         "`p` for the Dropout layer.")
```

```python
test_TorchDeepNeuralClassifier(TorchDeepNeuralClassifier)
```

### Your original system [3 points]

This is a simple dataset, but its "word-disjoint" nature ensures that it's a challenging one, and there are lots of modeling strategies one might adopt.

You are free to do whatever you like. We require only that your system differ in some way from those defined in the preceding questions. They don't have to be completely different, though. For example, you might want to stick with the model but represent examples differently, or the reverse.

You are free to use different pretrained word vectors and the like.

Please embed your code in this notebook so that we can rerun it.

In the cell below, please provide a brief technical description of your original system, so that the teaching team can gain an understanding of what it does. This will help us to understand your code and analyze all the submissions to identify patterns and strategies.  We also ask that you report the best score your system got during development, just to help us understand how systems performed overall.

```python
# PLEASE MAKE SURE TO INCLUDE THE FOLLOWING BETWEEN THE START AND STOP COMMENTS:
#   1) Textual description of your system.
#   2) The code for your original system.
#   3) The score achieved by your system in place of MY_NUMBER.
#        With no other changes to that line.
#        You should report your score as a decimal value <=1.0
# PLEASE MAKE SURE NOT TO DELETE OR EDIT THE START AND STOP COMMENTS

# NOTE: MODULES, CODE AND DATASETS REQUIRED FOR YOUR ORIGINAL SYSTEM
# SHOULD BE ADDED BELOW THE 'IS_GRADESCOPE_ENV' CHECK CONDITION. DOING
# SO ABOVE THE CHECK MAY CAUSE THE AUTOGRADER TO FAIL.

# START COMMENT: Enter your system description in this cell.
# The system uses:
#    The Glove 50 embedded space used previously as we didn't find any significant improvement in using other
# embedded spaces.
#
#    vec_concatenate to combine the 2 vector words in a single representation. If the hypothesis belongs to the
# premise hypernyms, hyponym, member_holonyms, antonyms, derivationally_related_forms, pertainyms as defined
# by Wordnet, we translate the relevant representation by a value which is halfway between the
# mean of the sums of the rows of the Glove embedded space and its max. For that purpose we split evenly a 50 array
# between these features. So that we have a 150 array, 50 for word1, 50 for word 2 and 50 for the different wordnet
# features.
#
#  Finally, we use a 3 hidden layers neural network classifier defined in the class TorchDeepNCMultipleLayers.
#
# My peak score was: 0.679

if 'IS_GRADESCOPE_ENV' not in os.environ:
    from nltk.corpus import wordnet as wn
    import torch.nn as nn

    utils.fix_random_seeds()

    class TorchDeepNCMultipleLayers(TorchShallowNeuralClassifier):

        def __init__(self, nhidden=1, **kwargs):
            """
            Generalisation of TorchShallowNeuralClassifier with multiple hidden layers. Each hidden layers keeps the
            dimension of the hidden layer has defined in the super class.
            :param nhidden: number of hidden layers
            """
            super().__init__(**kwargs)
            self.nhidden = nhidden

        def build_graph(self):
            """
            Returns
            -------
            an `nn.Module` instance, which can be a free-standing class you
            write yourself, as in `torch_rnn_classifier`, or the output of
            `nn.Sequential`, as in `torch_shallow_neural_classifier`.

            """
            graph_list = [nn.Linear(self.input_dim, self.hidden_dim)]
            graph_list.append(self.hidden_activation)
            for i in range(self.nhidden - 1):
                graph_list.append(nn.Linear(self.hidden_dim, self.hidden_dim))
                graph_list.append(self.hidden_activation)
            graph_list.append(nn.Linear(self.hidden_dim, self.n_classes_))
            graph = nn.Sequential(*tuple(graph_list))
            return graph

    def wordnet_features(word1, word2, methodname):
        """
        Returns 1 if the synsets extracted from the hierarchy methodname of the premise intersect the synsets of the
        hypothesis. Otherwise returns -1
        :param word1: word in premise
        :param word2: word in hypothesis
        :param methodname: wordnet hierarchy considered (hypernym or hyponym)
        :return:
        """
        try:
            hyps = [h for ss in wn.synsets(word1) for h in getattr(ss, methodname)()]
        except AttributeError:
            hyps = list()
            for ss in wn.synsets(word1):
                try:
                    for h in getattr(ss.lemmas()[0], methodname)():
                        hyps.append(h)
                except TypeError:
                    pass
        syns = wn.synsets(word2)

        output = 1 if set(hyps) & set(syns) else -1
        return output

    def word_entail_featurize_wordnet_1(data, vector_func, vector_combo_func, hierarchy, adjustment_param):
        """
        Modified version of the featurize function used in nli.wordentail_experiment. Uses WordNet hierarchy.
        :param data:
        :param vector_func:
        :param vector_combo_func:
        :param hierarchy: hierarchy in wordnet considered, 'hypernyms' or 'hyponyms'
        :param adjustment_param: Parameters by which we translate the vector output from vector_combo_func when
        the hypothesis is included in the wordnet hierarchy of word1 (hypernyms or hyponyms)
        :return:
        """
        X = []
        y = []
        for (w1, w2), label in data:
            is_related = list()
            for h in hierarchy:
                is_related.append(wordnet_features(w1, w2, h))
            rep = vector_combo_func(vector_func(w1), vector_func(w2))

            # build a vector of length vector_func(w1) with info from wordnet
            vec_size = vector_func(w1).shape[0]
            hierarchy_size = vec_size // len(hierarchy)
            coeff_vec = np.array([x for item in is_related for x in [item] * hierarchy_size])
            remaining_items = vec_size % len(hierarchy)
            if remaining_items > 0:
                # makes sure that coeff_vec has the right size
                coeff_vec = np.concatenate((coeff_vec, np.zeros(remaining_items)))
            adj_vec = coeff_vec * adjustment_param

            rep = np.concatenate((rep, adj_vec))
            X.append(rep)
            y.append(label)
        return X, y

    def word_entail_featurize_wordnet(hierarchy, param):
        """
        Wrapper of word_entail_featurize_wordnet_1, allowing to define parameters and adjustment_param
        in the feature function.
        """
        return lambda *args, **kwargs: word_entail_featurize_wordnet_1(*args, hierarchy=hierarchy,
                                                                   adjustment_param=param, **kwargs)

    def nli_model():
        """
        Original system
        """
        glove_dic = GLOVE
        df = pd.DataFrame.from_dict(glove_dic, orient='index')
        sum_rows = df.sum(axis=1)

        # calculate param, average distance between the mean and the max/min sum of row values of the Glove embedded space.
        # will be used in word_entail_featurize_wordnet to adjust vectors when they share an hypernym.
        n = df.shape[1]
        mean, max1, min1 = sum_rows.mean() / n, sum_rows.max() / n, sum_rows.min() / n
        param = 0.25 * (abs(min1) + abs(max1)) + mean

        featurize_func = word_entail_featurize_wordnet(['hypernyms', 'hyponyms', 'member_holonyms', 'antonyms', 'antonyms',
                                                        'derivationally_related_forms', 'pertainyms'], param)
        model = TorchDeepNCMultipleLayers(hidden_dim=150, nhidden=3, batch_size=300, eta=0.001, hidden_activation=nn.Tanh())

        experiment = nli.wordentail_experiment(
            train_data=wordentail_data['train'],
            assess_data=wordentail_data['dev'],
            model=model,
            vector_func=glove_vec,
            vector_combo_func=vec_concatenate,
            featurize_func=featurize_func)
        return experiment

    model_result = nli_model()

# STOP COMMENT: Please do not remove this comment.
```

    Stopping after epoch 73. Training loss did not improve more than tol=1e-05. Final error is 0.15200649778125808.

                  precision    recall  f1-score   support

               0      0.888     0.931     0.909      1732
               1      0.524     0.392     0.449       334

        accuracy                          0.844      2066
       macro avg      0.706     0.662     0.679      2066
    weighted avg      0.829     0.844     0.835      2066

## Bake-off [1 point]

The goal of the bake-off is to achieve the highest __macro-average F1__ score on a test set that we will make available at the start of the bake-off. The announcement will go out on the discussion forum. To enter, you'll be asked to run `nli.bake_off_evaluation` on the output of your chosen `nli.wordentail_experiment` run.

The cells below this one constitute your bake-off entry.

The rules described in the [Your original system](#Your-original-system-[3-points]) homework question are also in effect for the bake-off.

Systems that enter will receive the additional homework point, and systems that achieve the top score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.

Late entries will be accepted, but they cannot earn the extra 0.5 points. Similarly, you cannot win the bake-off unless your homework is submitted on time.

The announcement will include the details on where to submit your entry.

```python
# Enter your bake-off assessment code into this cell.
# Please do not remove this comment.
if 'IS_GRADESCOPE_ENV' not in os.environ:
    pass
    # Please enter your code in the scope of the above conditional.
    ##### YOUR CODE HERE

```

```python
# On an otherwise blank line in this cell, please enter
# your macro-avg f1 value as reported by the code above.
# Please enter only a number between 0 and 1 inclusive.
# Please do not remove this comment.
if 'IS_GRADESCOPE_ENV' not in os.environ:
    pass
    # Please enter your score in the scope of the above conditional.
    ##### YOUR CODE HERE

```


Wiki Game Bot

This notebook steps through a program designed to find an optimal solution for The Wiki Game, where players compete to find the shortest path between 2 random wikipedia articles by clicking links within the page.
Our strategy is to use GloVe word vectors to pick the links that match the target link the best. To measure similarity between links we are using the cosine similarity formula.

\begin{equation}
cos(a, b) = \frac{ab}{\left \| a \right \|\left \| b \right \|}
\end{equation}

The perfect solution could be computed by checking every link on every page for the target link, however this is exponentially computationally expensive. To find a balance between computing speed and the depth of search we are using a proccess inspired by Beam Search. By only searching the top n links (with the highest cosine similarity) the program is generally able to find a solution quickly. The program has certain limitations due to its use of the word embeddings. If the words of an article name aren't in the embedding dictionary then it's unable to converge on the target. If an article is a person, the embeddings are limited with names and may only converge on celebrities and famous names.

Import Statements

Numpy is used to handle the formatting & multiplication of the GloVe word embedding

urllib is used for opening and retrieving data from Wikipedia URLs

re (Regular Expressions) is used for searching and replacing parts of strings (links)

BeautifulSoup is used for parsing the HTML anchors of from Wikipedia Pages

```python
import numpy as np
import re
from bs4 import BeautifulSoup
from urllib.request import urlopen
```

```python
def create_embeddings():
    global embeddings_index
    embeddings_index = {}

    #Open the 300 dimensional embedding as a readable file
    with open('glove.6B.300d.txt', 'r', encoding='utf-8') as f:
        #Loop Through Each Line
        for line in f:
            #Lower & Strip the line, then split into a list
            glove_parsing = line.lower().strip().split()

            #Take the first element (which is a word)
            embedding_word = glove_parsing[0]

            #Then take the embedding weights and place them into an array
            embedding_weights = np.array(glove_parsing[1:])

            #Create a dictionary where keys correspond to a given word and the values are the given weights
            embeddings_index[embedding_word] = embedding_weights
    return embeddings_index
```

```python
def retrive_wiki_links(url):
    """Given a string input, url, returns a list of all blue article links"""

    #Retrive the Entire html of the Wiki Page and Convert to a String from Bytes
    html = str(urlopen(url).read())

    #Find the Start of the Content Page on the html
    index_start = html.index('id="mw-content-text"')

    #Find Where the End of the Content Page is, this is Usually Where the Reference Header Begins
    #By Doing this we also Avoid Picking up the Links in the Reference Section
    index_end = None
    if ('id="References"') in html:
        index_end = html.index('id="References"')
    #Sometimes a Note Secection Exists, Which Comes Before the Reference Section, which also Contains Links we Wish to Avoid
    if ('id="Notes"') in html:
        index_end = html.index('id="Notes"')

    #Concatenate the html to only include the Content Section
    html_clean = html[:index_start+1] + html[:index_end]

    #Process the html data using the Beautiful Soup library
    cleaned_data = BeautifulSoup(html_clean, 'html.parser')

    #Retrive a list of all anchor tags
    tags = cleaned_data('a')

    #Retrive href links
    tag_list = []
    for tag in tags:
        tag = tag.get('href')
        tag_list.append(tag)

    #Remvoe None type elements
    tag_list = [x for x in tag_list if x != None]

    #Loop through the tags and filter
    wiki_links = []
    for tag in tag_list:
        #Blue links have no unique identifiers compared to other links, therefore other link types must be removed
        removed_terms = ['.jpg', '.jpeg', '/User:', '/wiki/Talk', '.png', '/wiki/Wikipedia',
                         '/wiki/Help', '/wiki/File', '/wiki/Template', '/wiki/Special', 'https://',
                         'disambiguation']
        if ('/wiki/') in tag and not any(substring in tag for substring in removed_terms):
            wiki_links.append(tag)
    return wiki_links
```

```python
def wiki_link_cleaner(link):
    """Clean the raw link into words readable by the encodings"""

    link = re.sub('^(.*?)\/wiki\/', '', link)
    link = link.replace('_', ' ').split('#')[0]
    link = re.sub(r'[^\w\s]',' ', link)
    return link
```

```python
def words_to_embeddings(wiki_links):
    """given a list of links(wiki_links), return them as the embedding of the words"""

    #Strip the links down to the keywords
    clean_links = []
    for link in wiki_links:
        link = wiki_link_cleaner(link)
        clean_links.append(link)

    #Loop through the links to make embeddings
    embedding_tags = []
    for link in clean_links:
        words = link.split()

        #If the word isn't within the embeddings, it is removed
        for word in words:
            if embeddings_index.get(word.lower()) is None:
                words.remove(word)

        #Create a 2d array (length of words by the dimesnion of the word embedding)
        word_vecs = np.zeros((len(words), 300))

        #Add the respective weights to the array
        for i in range(len(words)):
            word_vecs[i] =  embeddings_index.get(words[i].lower())

        #Take the mean of the weights (used for multi word links)
        #It's possible to use a more complex method of multi-word embeddings
        embedding_tags.append(np.mean(word_vecs, axis=0))

    embedding_tags = np.array(embedding_tags)
    return embedding_tags
```

```python
#Function used to determine the similarity between words
def cosine_similarity(a, b):
    """Calulates the cosine similarity between vectors a and b"""

    numerator = np.dot(a, b)
    denominator = np.dot(np.linalg.norm(a), np.linalg.norm(b))
    return numerator / denominator

```

```python

def beam_search(target, embedding_tags, BEAM_VALUE):
    """Uses the principles of a beam search algorithm to find the best path to a range of new links"""

    tag_values = []
    target = target.lower()
    words = target.split()

    #If the word isn't within the embeddings, it is removed
    for word in words:
        if embeddings_index.get(word) is None:
            words.remove(word)

    #Create a 2d array (length of words by the dimesnion of the word embedding)
    word_vecs = np.zeros((len(words), 300))

    #Add the respective weights to the array
    for i in range(len(words)):
        word_vecs[i] =  embeddings_index.get(words[i])

    #Take the mean of the weights (used for multi word links)
    target_embedding = np.mean(word_vecs, axis=0)

    #Calculate the cosine similarities
    for i in range(len(embedding_tags)):
        tag_values.append(cosine_similarity(embedding_tags[i], target_embedding))

    for j in range(len(tag_values)):
        if np.isnan(tag_values[j]):
            tag_values[j] = 0

    #Beam value adjusted if is larger than the number of values
    temp = 0
    if len(tag_values) < BEAM_VALUE:
        temp = BEAM_VALUE
        BEAM_VALUE = len(tag_values)

    #Output the top BEAM_VALUE number of links
    output = np.argpartition(np.array(tag_values), -BEAM_VALUE)[-BEAM_VALUE:]

    BEAM_VALUE = temp
    return output

```

```python
def dirty_to_searchable(index, word_list, return_link = False):
    """takes a half link and makes it a full wikipedia link"""

    index = int(index)
    link = word_list[index]
    if return_link == True:
        link = 'https://en.wikipedia.org'+link
    return link
```

```python
def searchable_to_dirty(link):
    """removes the wikipedia part of the link"""
    return link.split(".wikipedia.org")[-1]
```

```python
def wiki_game(starting_link, target_link):
    target = wiki_link_cleaner(target_link)
    blacklist = []
    link_position = {}
    number_of_searches = 0
    searches = []
    searches_history = []
    beam = 8

    #initial page search
    wiki_links = retrive_wiki_links(starting_link)

    #Create embeddings of the links
    embedding_tags = words_to_embeddings(wiki_links)

    #Run beam search to find the best embeddings (best 12 links on the first page)
    beam_initial_positions = beam_search(target, embedding_tags, 12)

    #Format new links from the beam search
    for i in beam_initial_positions:
        searches.append(dirty_to_searchable(i, wiki_links, True))
    searches_history.append(searches)

    #Keep checking new links until the target_link is found
    print('This may take a minute...')
    while target_link not in searches:
        positions_beam_1 = []
        total_links_beam_1 = []
        total_links_beam_2 = []
        number_of_searches += 1

        #Limit the number of searches to prevent an infinite loop
        if number_of_searches > 9:
            print('Max search limit exceeded')
            is_successful = False
            break

        #Filter out links that have already been used to prevent looping
        for i in searches:
            wiki_links = retrive_wiki_links(i)
            for wiki_link in wiki_links:
                while wiki_links.count(wiki_link) != 1:
                    wiki_links.remove(wiki_link)
                if wiki_link in blacklist:
                    wiki_links.remove(wiki_link)
                else:
                    total_links_beam_1.append(wiki_link)
                    blacklist.append(wiki_link)

        for x in total_links_beam_1:
            while total_links_beam_1.count(x) != 1:
                total_links_beam_1.remove(x)

        #Beam search starts by picking top 8 links, and increases its beam by 3 each iteration
        embedding_tags = words_to_embeddings(total_links_beam_1)
        positions_beam_1 = beam_search(target, embedding_tags, beam)
        beam += 3

        #format the new links and add them to searches_history for the next iteration
        searches = []
        for position in positions_beam_1:
            link = dirty_to_searchable(position, total_links_beam_1, True)
            searches.append(link)
        searches_history.append(searches)
        print('searching...')

    return searches_history, starting_link, target_link
```

```python

def track_link_path(searches_history, starting_link, target_link):
    """Works backwards from the solution, using searches_history to find a path to the starting_link"""

    path = []
    term = searchable_to_dirty(target_link)
    path.append(target_link)
    for j in range(len(searches_history))[:-1][::-1]:
        for i in searches_history[j]:
            wiki_links = retrive_wiki_links(i)
            for link in wiki_links:
                if link == term:
                    path.append(i)
                    term = searchable_to_dirty(i)
                    break
            else:
                continue
            break
    path.append(starting_link)
    return path[::-1]

```

```python
embeddings_index = create_embeddings() #Takes a minute to run
```

```python
starting_link = input('Starting Link --- ')
target_link = input('Target Link --- ')

searches_history, starting_link, target_link = wiki_game(starting_link, target_link)
path = track_link_path(searches_history, starting_link, target_link)

for i in range(len(path)-1):
    print(path[i], end = " ---> ")
print(path[-1])

```

    Starting Link --- https://en.wikipedia.org/wiki/Apple_pie
    Target Link --- https://en.wikipedia.org/wiki/Shrek
    This may take a minute...
    searching...
    searching...
    searching...
    https://en.wikipedia.org/wiki/Apple_pie ---> https://en.wikipedia.org/wiki/Ginger ---> https://en.wikipedia.org/wiki/Gingerbread ---> https://en.wikipedia.org/wiki/Gingerbread_man ---> https://en.wikipedia.org/wiki/Shrek

```python
starting_link = 'https://en.wikipedia.org/wiki/Interference_(communication)'
target_link = 'https://en.wikipedia.org/wiki/Popcorn'

searches_history, starting_link, target_link = wiki_game(starting_link, target_link)
path = track_link_path(searches_history, starting_link, target_link)

for i in range(len(path)-1):
    print(path[i], end = " ---> ")
print(path[-1])

```

    This may take a minute...
    searching...
    searching...
    searching...
    https://en.wikipedia.org/wiki/Interference_(communication) ---> https://en.wikipedia.org/wiki/Wireless_networks ---> https://en.wikipedia.org/wiki/Wi-Fi ---> https://en.wikipedia.org/wiki/Microwave_oven ---> https://en.wikipedia.org/wiki/Popcorn

```python
starting_link = 'https://en.wikipedia.org/wiki/Early_Christianity'
target_link = 'https://en.wikipedia.org/wiki/Eminem'

searches_history, starting_link, target_link = wiki_game(starting_link, target_link)
path = track_link_path(searches_history, starting_link, target_link)

for i in range(len(path)-1):
    print(path[i], end = " ---> ")
print(path[-1])
```

    This may take a minute...
    searching...
    searching...
    https://en.wikipedia.org/wiki/Early_Christianity ---> https://en.wikipedia.org/wiki/Christian_music ---> https://en.wikipedia.org/wiki/Music_download ---> https://en.wikipedia.org/wiki/Eminem

```python
starting_link = 'https://en.wikipedia.org/wiki/Apple_pie'
target_link = 'https://en.wikipedia.org/wiki/Shrek'

searches_history, starting_link, target_link = wiki_game(starting_link, target_link)
path = track_link_path(searches_history, starting_link, target_link)

for i in range(len(path)-1):
    print(path[i], end = " ---> ")
print(path[-1])
```

    This may take a minute...
    searching...
    searching...
    searching...
    https://en.wikipedia.org/wiki/Apple_pie ---> https://en.wikipedia.org/wiki/Ginger ---> https://en.wikipedia.org/wiki/Gingerbread ---> https://en.wikipedia.org/wiki/Gingerbread_man ---> https://en.wikipedia.org/wiki/Shrek


# Orthogonal Random Forest: Use Cases and Examples

Orthogonal Random Forest (ORF) combines orthogonalization,
a technique that effectively removes the confounding effect in two-stage estimation,
with generalized random forests, a flexible method for estimating treatment effect heterogeneity. Due to the orthogonalization aspect of this method, the ORF performs especially well in the presence of high-dimensional confounders. For more details, see [this paper](https://arxiv.org/abs/1806.03467).

The EconML SDK implements the following OrthoForest variants:

* ContinuousTreatmentOrthoForest: suitable for continuous treatments

* DiscreteTreatmentOrthoForest: suitable for discrete treatments

In this notebook, we show the performance of the ORF on synthetic data.

**Notebook contents:**

1. Example usage with continuous treatment synthetic data

2. Example usage with binary treatment synthetic data

3. Example usage with multiple discrete treatment synthetic data

4. Example usage with real continuous treatment observational data

```python
import econml
```

```python
# Main imports
from econml.ortho_forest import ContinuousTreatmentOrthoForest, DiscreteTreatmentOrthoForest
from econml.sklearn_extensions.linear_model import WeightedLassoCVWrapper, WeightedLasso, WeightedLassoCV

# Helper imports
import numpy as np
from itertools import product
from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, LogisticRegressionCV
import matplotlib.pyplot as plt

%matplotlib inline
```

## 1. Example Usage with Continuous Treatment Synthetic Data

### 1.1. DGP
We use the data generating process (DGP) from [here](https://arxiv.org/abs/1806.03467). The DGP is described by the following equations:

\begin{align}
T =& \langle W, \beta\rangle + \eta, & \;\eta \sim \text{Uniform}(-1, 1)\\
Y =& T\cdot \theta(X) + \langle W, \gamma\rangle + \epsilon, &\; \epsilon \sim \text{Uniform}(-1, 1)\\
W \sim& \text{Normal}(0,\, I_{n_w})\\
X \sim& \text{Uniform}(0,1)^{n_x}
\end{align}

where $W$ is a matrix of high-dimensional confounders and $\beta, \gamma$ have high sparsity.

For this DGP,
\begin{align}
\theta(x) = \exp(2\cdot x_1).
\end{align}

```python
# Treatment effect function
def exp_te(x):
    return np.exp(2*x[0])
```

```python
# DGP constants
np.random.seed(123)
n = 1000
n_w = 30
support_size = 5
n_x = 1
# Outcome support
support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
coefs_Y = np.random.uniform(0, 1, size=support_size)
epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
# Treatment support
support_T = support_Y
coefs_T = np.random.uniform(0, 1, size=support_size)
eta_sample = lambda n: np.random.uniform(-1, 1, size=n)

# Generate controls, covariates, treatments and outcomes
W = np.random.normal(0, 1, size=(n, n_w))
X = np.random.uniform(0, 1, size=(n, n_x))
# Heterogeneous treatment effects
TE = np.array([exp_te(x_i) for x_i in X])
T = np.dot(W[:, support_T], coefs_T) + eta_sample(n)
Y = TE * T + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)

# ORF parameters and test data
# The following parameters are set according to theory
subsample_power = 0.88
subsample_ratio = ((n/np.log(n_w))**(subsample_power)) / n
lambda_reg = np.sqrt(np.log(n_w) / (10 * subsample_ratio * n))
X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))
```

### 1.2. Train Estimator

**Note:** The models in the final stage of the estimation (``model_T_final``, ``model_Y_final``) need to support sample weighting.

If the models of choice do not support sample weights (e.g. ``sklearn.linear_model.LassoCV``), the ``econml`` packages provides a convenient wrapper for these models ``WeightedModelWrapper`` in order to allow sample weights.

If the model of choice is a linear (regression) model such as Lasso, you should set ``sample_type="weighted"``. Otherwise, set ``sample_type="sampled"``.

```python
est = ContinuousTreatmentOrthoForest(
    n_trees=200, min_leaf_size=5,
    max_depth=50, subsample_ratio=2*subsample_ratio, bootstrap=False,
    model_T=Lasso(alpha=lambda_reg),
    model_Y=Lasso(alpha=lambda_reg),
    model_T_final=WeightedLasso(alpha=lambda_reg),
    model_Y_final=WeightedLasso(alpha=lambda_reg),
    random_state=123)
```

```python
est.fit(Y, T, X, W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    4.3s
    [Parallel(n_jobs=-1)]: Done 185 out of 200 | elapsed:    5.8s remaining:    0.4s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    5.8s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.1s
    [Parallel(n_jobs=-1)]: Done 118 out of 200 | elapsed:    1.0s remaining:    0.7s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    1.4s finished

    <econml.ortho_forest.ContinuousTreatmentOrthoForest at 0x1f2709b4898>

```python
treatment_effects = est.effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.4s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    2.2s finished

### 1.3. Performance Visualization

```python
y = treatment_effects[:, 0]
plt.plot(X_test, y, label='ORF estimate')
expected_te = np.array([exp_te(x_i) for x_i in X_test])
plt.plot(X_test[:, 0], expected_te, 'b--', label='True effect')
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

## 2. Example Usage with Binary Treatment Synthetic Data

### 2.1. DGP
We use the following DGP:

\begin{align}
T \sim & \text{Bernoulli}\left(f(W)\right), &\; f(W)=\sigma(\langle W, \beta\rangle + \eta), \;\eta \sim \text{Uniform}(-1, 1)\\
Y = & T\cdot \theta(X) + \langle W, \gamma\rangle + \epsilon, & \; \epsilon \sim \text{Uniform}(-1, 1)\\
W \sim & \text{Normal}(0,\, I_{n_w}) & \\
X \sim & \text{Uniform}(0,\, 1)^{n_x}
\end{align}

where $W$ is a matrix of high-dimensional confounders, $\beta, \gamma$ have high sparsity and $\sigma$ is the sigmoid function.

For this DGP,
\begin{align}
\theta(x) = \exp( 2\cdot x_1 ).
\end{align}

```python
# DGP constants
np.random.seed(1234)
n = 1000
n_w = 30
support_size = 5
n_x = 1
# Outcome support
support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
coefs_Y = np.random.uniform(0, 1, size=support_size)
epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
# Treatment support
support_T = support_Y
coefs_T = np.random.uniform(0, 1, size=support_size)
eta_sample = lambda n: np.random.uniform(-1, 1, size=n)

# Generate controls, covariates, treatments and outcomes
W = np.random.normal(0, 1, size=(n, n_w))
X = np.random.uniform(0, 1, size=(n, n_x))
# Heterogeneous treatment effects
TE = np.array([exp_te(x_i) for x_i in X])
# Define treatment
log_odds = np.dot(W[:, support_T], coefs_T) + eta_sample(n)
T_sigmoid = 1/(1 + np.exp(-log_odds))
T = np.array([np.random.binomial(1, p) for p in T_sigmoid])
# Define the outcome
Y = TE * T + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)

# ORF parameters and test data
# The following parameters are set according to theory
subsample_power = 0.88
subsample_ratio = ((n/np.log(n_w))**(subsample_power)) / n
lambda_reg = np.sqrt(np.log(n_w) / (10 * subsample_ratio * n))
X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))
```

### 2.2. Train Estimator

```python
est = DiscreteTreatmentOrthoForest(
    n_trees=200, min_leaf_size=10,
    max_depth=30, subsample_ratio=2*subsample_ratio, bootstrap=False,
    propensity_model = LogisticRegression(C=1/(X.shape[0]*lambda_reg), penalty='l1', solver='saga'),
    model_Y = Lasso(alpha=lambda_reg),
    propensity_model_final=LogisticRegression(C=1/(X.shape[0]*lambda_reg), penalty='l1', solver='saga'),
    model_Y_final=WeightedLasso(alpha=lambda_reg)
)
```

```python
est.fit(Y, T, X, W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.1s
    [Parallel(n_jobs=-1)]: Done 118 out of 200 | elapsed:    0.8s remaining:    0.5s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    1.2s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.1s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    1.1s finished

    <econml.ortho_forest.DiscreteTreatmentOrthoForest at 0x1f2754e17f0>

```python
treatment_effects = est.effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.4s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.2s finished

### 2.3. Performance Visualization

```python
y = treatment_effects
plt.plot(X_test, y, label='ORF estimate')
expected_te = np.array([exp_te(x_i) for x_i in X_test])
plt.plot(X_test[:, 0], expected_te, 'b--', label='True effect')
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

## 3. Example Usage with Multiple Treatment Synthetic Data

### 3.1 DGP
We use the following DGP:

\begin{align}
Y = & \sum_{t=1}^{n_{\text{treatments}}} 1\{T=t\}\cdot \theta_{T}(X) + \langle W, \gamma\rangle + \epsilon, \; \epsilon \sim \text{Unif}(-1, 1), \\
\text{Pr}[T=t \mid W] \propto & \exp\{\langle W, \beta_t \rangle\}, \;\;\;\; \forall t\in \{0, 1, \ldots, n_{\text{treatments}}\}
\end{align}

where $W$ is a matrix of high-dimensional confounders, $\beta_t, \gamma$ are sparse.

For this particular example DGP we used $n_{\text{treatments}}=3$ and
\begin{align}
\theta_1(x) = & \exp( 2 x_1 ),\\
\theta_2(x) = &  3 \cdot \sigma(100\cdot (x_1 - .5)),\\
\theta_3(x) = & -2 \cdot \sigma(100\cdot (x_1 - .25)),
\end{align}
where $\sigma$ is the sigmoid function.

```python
def get_test_train_data(n, n_w, support_size, n_x, te_func, n_treatments):
    # Outcome support
    support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
    coefs_Y = np.random.uniform(0, 1, size=support_size)
    epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
    # Treatment support
    support_T = support_Y
    coefs_T = np.random.uniform(0, 1, size=(support_size, n_treatments))
    eta_sample = lambda n: np.random.uniform(-1, 1, size=n)
    # Generate controls, covariates, treatments and outcomes
    W = np.random.normal(0, 1, size=(n, n_w))
    X = np.random.uniform(0, 1, size=(n, n_x))
    # Heterogeneous treatment effects
    TE = np.array([te_func(x_i, n_treatments) for x_i in X])
    log_odds = np.dot(W[:, support_T], coefs_T)
    T_sigmoid = np.exp(log_odds)
    T_sigmoid = T_sigmoid/np.sum(T_sigmoid, axis=1, keepdims=True)
    T = np.array([np.random.choice(n_treatments, p=p) for p in T_sigmoid])
    TE = np.concatenate((np.zeros((n,1)), TE), axis=1)
    Y = TE[np.arange(n), T] + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)
    X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))

    return (Y, T, X, W), (X_test, np.array([te_func(x, n_treatments) for x in X_test]))
```

```python
import scipy.special
def te_func(x, n_treatments):
    return [np.exp(2*x[0]), 3*scipy.special.expit(100*(x[0] - .5)) - 1, -2*scipy.special.expit(100*(x[0] - .25))]

np.random.seed(123)
(Y, T, X, W), (X_test, te_test) = get_test_train_data(1000, 3, 3, 1, te_func, 4)
```

### 3.2 Train Estimator

```python
est = DiscreteTreatmentOrthoForest(n_trees=500,
                                   model_Y = WeightedLasso(alpha=lambda_reg))
```

```python
est.fit(Y, T, X, W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.3s
    [Parallel(n_jobs=-1)]: Done 208 tasks      | elapsed:    3.7s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:    9.1s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.4s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    2.2s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:    5.4s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:    9.9s finished

    <econml.ortho_forest.DiscreteTreatmentOrthoForest at 0x1f2730a2f60>

```python
treatment_effects = est.const_marginal_effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.9s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    5.1s finished

### 3.3 Performance Visualization

```python
y = treatment_effects
for it in range(y.shape[1]):
    plt.plot(X_test, y[:, it], label='ORF estimate T={}'.format(it))
    plt.plot(X_test[:, 0], te_test[:, it], '--', label='True effect T={}'.format(it))
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

## 4. Example usage with real continuous treatment observational data

We applied our technique to Dominick’s dataset, a popular historical dataset of store-level orange juice prices and sales provided by University of Chicago Booth School of Business.

The dataset is comprised of a large number of covariates $W$, but researchers might only be interested in learning the elasticity of demand as a function of a few variables $x$ such
as income or education.

We applied the `ContinuousTreatmentOrthoForest` to estimate orange juice price elasticity
as a function of income, and our results, unveil the natural phenomenon that lower income consumers are more price-sensitive.

### 4.1. Data

```python
# A few more imports
import os
import pandas as pd
import urllib.request
from sklearn.preprocessing import StandardScaler
```

```python
# Import the data
file_name = "oj_large.csv"

if not os.path.isfile(file_name):
    print("Downloading file (this might take a few seconds)...")
    urllib.request.urlretrieve("https://msalicedatapublic.blob.core.windows.net/datasets/OrangeJuice/oj_large.csv", file_name)
oj_data = pd.read_csv(file_name)
oj_data.head()
```

    Downloading file (this might take a few seconds)...

      store
      brand
      week
      logmove
      feat
      price
      AGE60
      EDUC
      ETHNIC
      INCOME
      HHLARGE
      WORKWOM
      HVAL150
      SSTRDIST
      SSTRVOL
      CPDIST5
      CPWVOL5

      0
      2
      tropicana
      40
      9.018695
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      1
      2
      tropicana
      46
      8.723231
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      2
      2
      tropicana
      47
      8.253228
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      3
      2
      tropicana
      48
      8.987197
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      4
      2
      tropicana
      50
      9.093357
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

```python
# Prepare data
Y = oj_data['logmove'].values
T = np.log(oj_data["price"]).values
scaler = StandardScaler()
W1 = scaler.fit_transform(oj_data[[c for c in oj_data.columns if c not in ['price', 'logmove', 'brand', 'week', 'store']]].values)
W2 = pd.get_dummies(oj_data[['brand']]).values
W = np.concatenate([W1, W2], axis=1)
X = oj_data[['INCOME']].values
```

### 4.2. Train Estimator

```python
# Define some parameters
n_trees = 2000
min_leaf_size = 50
max_depth = 20
subsample_ratio = 0.02
bootstrap = False
```

```python
est = ContinuousTreatmentOrthoForest(
        n_trees=n_trees, min_leaf_size=min_leaf_size, max_depth=max_depth,
        subsample_ratio=subsample_ratio, bootstrap=bootstrap,
        model_T=Lasso(alpha=0.1),
        model_Y=Lasso(alpha=0.1),
        model_T_final=WeightedLassoCVWrapper(cv=3),
        model_Y_final=WeightedLassoCVWrapper(cv=3)
       )
```

```python
est.fit(Y, T, X, W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done 296 tasks      | elapsed:    0.5s
    [Parallel(n_jobs=-1)]: Done 2000 out of 2000 | elapsed:    2.2s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.1s
    [Parallel(n_jobs=-1)]: Done 2000 out of 2000 | elapsed:    2.3s finished

    <econml.ortho_forest.ContinuousTreatmentOrthoForest at 0x1f27a765ac8>

```python
min_income = 10.0
max_income = 11.1
delta = (max_income - min_income) / 100
X_test = np.arange(min_income, max_income + delta - 0.001, delta).reshape(-1, 1)
```

```python
import time
t0 = time.time()
te_pred = est.const_marginal_effect(X_test)
print(time.time() - t0)
```

    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   41.0s

    285.2490632534027

    [Parallel(n_jobs=-1)]: Done 101 out of 101 | elapsed:  4.8min finished

### 4.3. Performance Visualization

```python
# Plot Orange Juice elasticity as a function of income
plt.plot(np.ndarray.flatten(X_test), te_pred[:, 0], label="OJ Elasticity")
plt.xlabel(r'$\log$(Income)')
plt.ylabel('Orange Juice Elasticity')
plt.legend()
plt.title("Orange Juice Elasticity vs Income")
plt.show()
```

### 4.4 Bootstrap Confidence Intervals

We can also use a bootstrap estimator to generate confidence intervals by changing how we call `fit`; in order to return results in a few minutes we're limiting the number of trees to 100 and the number of bootstrap samples to 10 in the code below, but for better estimates these numbers can be increased at the cost of increased runtime.

```python
from econml.inference import BootstrapInference
est = ContinuousTreatmentOrthoForest(
        n_trees=100, min_leaf_size=min_leaf_size, max_depth=max_depth,
        subsample_ratio=subsample_ratio, bootstrap=bootstrap,
        model_T=Lasso(alpha=0.1),
        model_Y=Lasso(alpha=0.1),
        model_T_final=WeightedLassoCVWrapper(cv=3),
        model_Y_final=WeightedLassoCVWrapper(cv=3)
)
```

```python
est.fit(Y, T, X, W, inference=BootstrapInference(n_bootstrap_samples=10, n_jobs=-1))
te_pred_interval = est.const_marginal_effect_interval(X_test, alpha=0.02)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    0.2s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    0.1s finished
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.3s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.2s finished
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.3s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.3s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.3s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.4s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.3s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.4s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.3s finished
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.2s
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.5s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.5s
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    2.4s finished
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.6s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.6s finished
    [Parallel(n_jobs=-1)]: Done   3 out of  10 | elapsed:    7.8s remaining:   18.4s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.6s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.5s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.5s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.5s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    3.5s finished
    [Parallel(n_jobs=-1)]: Done   7 out of  10 | elapsed:    8.0s remaining:    3.4s
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    1.9s finished
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.1s
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.1s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    0.5s finished
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.2s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    0.8s finished
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    0.6s finished
    [Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    9.6s finished
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.

```python
plt.plot(np.ndarray.flatten(X_test), te_pred[:, 0], label="OJ Elasticity")
plt.fill_between(np.ndarray.flatten(X_test),
                 te_pred_interval[0][:, 0],
                 te_pred_interval[1][:, 0], alpha=.5, label="1-99% CI")
plt.xlabel(r'$\log$(Income)')
plt.ylabel('Orange Juice Elasticity')
plt.title("Orange Juice Elasticity vs Income")
plt.legend()
plt.show()
```

```python

```

notebooks/Orthogonal Random Forest Examples.ipynb