# Predicting Chicago Taxi Pickup Density using Random Forest

## Importing libraries

```python
import os
os.chdir('/Users/Kevin/Documents/TAMIDS Taxi Data')

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import cross_val_score
from sklearn.metrics import mean_squared_error
from sklearn.metrics import accuracy_score
import random
import math
import datetime
import time
```

## Data

```python
x_train = pd.read_csv('PickupData.csv')
x_test = pd.read_csv('PickupDataTest.csv')
```

A preview of our data.

```python
x_train.head()
```

      Year_num
      Month
      Hour
      Day
      WDay
      Latitude
      Longitude
      Pickups
      Month_num
      Hour_num
      Day_num
      Year_sin
      Year_cos
      Month_sin
      Month_cos
      Hour_sin
      Hour_cos
      Day_sin
      Day_cos

      0
      0.202740
      3
      17
      15
      5
      41.879255
      -87.642649
      2
      0.466667
      0.750000
      0.678571
      0.956235
      0.292600
      0.207912
      -0.978148
      -1.000000
      -1.836970e-16
      -0.900969
      -0.433884

      1
      0.106849
      2
      19
      8
      5
      41.893216
      -87.637844
      1
      0.233333
      0.833333
      0.690476
      0.622047
      0.782980
      0.994522
      0.104528
      -0.866025
      5.000000e-01
      -0.930874
      -0.365341

      2
      0.339726
      5
      20
      4
      6
      41.916005
      -87.675095
      3
      0.100000
      0.875000
      0.839286
      0.845249
      -0.534373
      0.587785
      0.809017
      -0.707107
      7.071068e-01
      -0.846724
      0.532032

      3
      0.695890
      9
      12
      11
      3
      41.884987
      -87.620993
      4
      0.333333
      0.541667
      0.363095
      -0.942761
      -0.333469
      0.866025
      -0.500000
      -0.258819
      -9.659258e-01
      0.757972
      -0.652287

      4
      0.953425
      12
      21
      14
      6
      41.906026
      -87.675312
      2
      0.433333
      0.916667
      0.845238
      -0.288482
      0.957485
      0.406737
      -0.913545
      -0.500000
      8.660254e-01
      -0.826239
      0.563320

### Defining Time Variables
The time granularity with respect to the granularity of location will be defined as bins per day (hours per day).  We must adjust our timestamps accordingly. Note that our data was processed in R.
* **Hour_num**: Using a 24-hour clock, the hours will be defined from 1-24 (1:00am is 1, 12:00am is 24). Using the **strptime()** function, the hours are defined from 0-23, so we just add 1 to correct this. The formula for this calculation is:
\begin{equation}
HourNum = (Hour+1)/24
\end{equation}

* **Day_num**: The day of the week will be defined from 0-6 (Monday is 0, Sunday is 6), and we will incorporate hour of day in this calculation also. The formula for this calculation is:
\begin{equation}
DayNum = (((WDay-1) \mod 7)+(Hour+1)/24)/7
\end{equation}
**WDay** is the day of the week.
Example: Friday, 2:00pm (Day = 5, Hour = 14)
\begin{equation}
    DayNum = ((5 \mod 7)+(14+1)/24)/7
\end{equation}
We do **WDay**-1 to count all of the days of the week previous to Friday. Since our range of integers for week is defined on the interval [0,6] we will be doing calculations for day of the week on mod 7. Then finally we divide by 7, the number of days in a week.

* **Month_num**: The day of the month.
\begin{equation}
MonthNum = (Day-1)/30
\end{equation}

* **Year_num**: The day of the year will defined on a scale of 1-365. The **strptime()** function defines the day of the year from 0-364, so like hour, we add an correction factor of 1. The formula for this calculation is:
\begin{equation}
YearNum = (YearDay+1)/365
\end{equation}

Note: For 2016, we used 366 to count since it is a leap year.

* **Sine** and **Cosine**: We capture cyclicality by applying sine and cosine functions to our calculated time, month, day, and year numbers.

### Create response variables

```python
y_pickup_train = np.log10(x_train['Pickups']+1)
y_pickup_test = np.log10(x_test['Pickups']+1)
```

```python
# Remove pickup and wday column from training and testing data
pickup_train = x_train.drop(['Pickups', 'WDay'], axis=1)
pickup_test = x_test.drop(['Pickups', 'WDay'], axis=1)
```

## Model Building
### Optimizing the random forest hyper-parameters
Before we run our model, we want to find the best parameters to run the model with, in particular, we want to optimize the number of trees to use, the depth of the trees, and the number of variables considered at each split. We will use a 5-fold cross validation function to find the best parameters.

```python
def optimize_params(clf, params, X, y):
    cv = GridSearchCV(clf, param_grid=params, n_jobs=1, cv=5, verbose=3)
    cv.fit(X, y)
    print('Best parameters:', cv.best_params_)
    print('Best score:', cv.best_score_)
    print('Best estimators:', cv.best_estimator_)
```

#### Defining the range of parameters
Note that **n_estimators** is the number of trees, **max_features** is the number of variables considered at each split, and **max_depth** is the depth of the trees.

```python
rf_estimator = RandomForestRegressor(n_estimators=20, n_jobs=-1)

params = {'n_estimators': [1, 2, 5, 10, 30, 50, 100],
          'max_features': ['auto', 'log2'],
          'max_depth': [10, 20, 50, 100]
         }

start_time = time.time()
best_params = optimize_params(rf_estimator, params, pickup_train, y_pickup_train)
time.time() - start_time
```

    Fitting 5 folds for each of 56 candidates, totalling 280 fits
    [CV] max_depth=10, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=1, score=0.559440, total=   2.2s
    [CV] max_depth=10, max_features=auto, n_estimators=1 .................

    [Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    2.3s remaining:    0.0s

    [CV]  max_depth=10, max_features=auto, n_estimators=1, score=0.612251, total=   2.2s
    [CV] max_depth=10, max_features=auto, n_estimators=1 .................

    [Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    4.6s remaining:    0.0s

    [CV]  max_depth=10, max_features=auto, n_estimators=1, score=0.615317, total=   2.1s
    [CV] max_depth=10, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=1, score=0.611700, total=   2.1s
    [CV] max_depth=10, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=1, score=0.558787, total=   2.1s
    [CV] max_depth=10, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=2, score=0.565525, total=   2.6s
    [CV] max_depth=10, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=2, score=0.616767, total=   2.6s
    [CV] max_depth=10, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=2, score=0.620318, total=   2.6s
    [CV] max_depth=10, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=2, score=0.616743, total=   2.6s
    [CV] max_depth=10, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=2, score=0.565599, total=   2.5s
    [CV] max_depth=10, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=5, score=0.568239, total=   3.9s
    [CV] max_depth=10, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=5, score=0.619466, total=   3.9s
    [CV] max_depth=10, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=5, score=0.622651, total=   3.9s
    [CV] max_depth=10, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=5, score=0.618336, total=   4.0s
    [CV] max_depth=10, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=10, max_features=auto, n_estimators=5, score=0.569112, total=   3.9s
    [CV] max_depth=10, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=10, score=0.568881, total=   8.2s
    [CV] max_depth=10, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=10, score=0.620670, total=   8.2s
    [CV] max_depth=10, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=10, score=0.623815, total=   8.2s
    [CV] max_depth=10, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=10, score=0.619361, total=   8.2s
    [CV] max_depth=10, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=10, score=0.570462, total=   8.2s
    [CV] max_depth=10, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=30, score=0.569266, total=  21.4s
    [CV] max_depth=10, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=30, score=0.621259, total=  21.6s
    [CV] max_depth=10, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=30, score=0.624219, total=  21.5s
    [CV] max_depth=10, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=30, score=0.620186, total=  21.5s
    [CV] max_depth=10, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=30, score=0.570721, total=  21.7s
    [CV] max_depth=10, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=50, score=0.569768, total=  36.5s
    [CV] max_depth=10, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=50, score=0.621234, total=  36.9s
    [CV] max_depth=10, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=50, score=0.624242, total=  36.4s
    [CV] max_depth=10, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=50, score=0.620450, total=  36.6s
    [CV] max_depth=10, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=10, max_features=auto, n_estimators=50, score=0.570956, total=  37.0s
    [CV] max_depth=10, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=auto, n_estimators=100, score=0.569641, total= 1.2min
    [CV] max_depth=10, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=auto, n_estimators=100, score=0.621275, total= 1.2min
    [CV] max_depth=10, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=auto, n_estimators=100, score=0.624468, total= 1.2min
    [CV] max_depth=10, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=auto, n_estimators=100, score=0.620398, total=93.1min
    [CV] max_depth=10, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=auto, n_estimators=100, score=0.571184, total= 2.3min
    [CV] max_depth=10, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=1, score=0.466810, total=   2.3s
    [CV] max_depth=10, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=1, score=0.540672, total=   1.8s
    [CV] max_depth=10, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=1, score=0.544281, total=   1.7s
    [CV] max_depth=10, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=1, score=0.447568, total=   1.7s
    [CV] max_depth=10, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=1, score=0.539146, total=   1.7s
    [CV] max_depth=10, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=2, score=0.512723, total=   1.6s
    [CV] max_depth=10, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=2, score=0.551522, total=   1.7s
    [CV] max_depth=10, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=2, score=0.516677, total=   1.6s
    [CV] max_depth=10, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=2, score=0.590157, total=   1.7s
    [CV] max_depth=10, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=2, score=0.530462, total=   1.6s
    [CV] max_depth=10, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=5, score=0.550823, total=   2.2s
    [CV] max_depth=10, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=5, score=0.557880, total=   2.3s
    [CV] max_depth=10, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=5, score=0.583998, total=   2.4s
    [CV] max_depth=10, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=5, score=0.581069, total=   2.3s
    [CV] max_depth=10, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=10, max_features=log2, n_estimators=5, score=0.555684, total=   2.2s
    [CV] max_depth=10, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=10, score=0.555615, total=   4.5s
    [CV] max_depth=10, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=10, score=0.569776, total=   4.5s
    [CV] max_depth=10, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=10, score=0.575308, total=   4.5s
    [CV] max_depth=10, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=10, score=0.584738, total=   4.6s
    [CV] max_depth=10, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=10, score=0.546476, total=   5.2s
    [CV] max_depth=10, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=30, score=0.558156, total=  11.2s
    [CV] max_depth=10, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=30, score=0.581067, total=  10.7s
    [CV] max_depth=10, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=30, score=0.572669, total=  11.6s
    [CV] max_depth=10, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=30, score=0.590424, total=  11.4s
    [CV] max_depth=10, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=30, score=0.568443, total=  11.5s
    [CV] max_depth=10, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=50, score=0.556932, total=  18.0s
    [CV] max_depth=10, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=50, score=0.588406, total=  19.2s
    [CV] max_depth=10, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=50, score=0.585809, total=  18.0s
    [CV] max_depth=10, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=50, score=0.594116, total=  18.3s
    [CV] max_depth=10, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=10, max_features=log2, n_estimators=50, score=0.570216, total=  19.1s
    [CV] max_depth=10, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=log2, n_estimators=100, score=0.558604, total=  35.2s
    [CV] max_depth=10, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=log2, n_estimators=100, score=0.590537, total=  35.5s
    [CV] max_depth=10, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=log2, n_estimators=100, score=0.585481, total=  35.4s
    [CV] max_depth=10, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=log2, n_estimators=100, score=0.589673, total=  36.4s
    [CV] max_depth=10, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=10, max_features=log2, n_estimators=100, score=0.566365, total=  35.0s
    [CV] max_depth=20, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=1, score=0.432941, total=   9.7s
    [CV] max_depth=20, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=1, score=0.629447, total=   8.7s
    [CV] max_depth=20, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=1, score=0.637581, total=   8.1s
    [CV] max_depth=20, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=1, score=0.517900, total=   6.5s
    [CV] max_depth=20, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=1, score=0.365630, total=   5.5s
    [CV] max_depth=20, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=2, score=0.528992, total=   6.0s
    [CV] max_depth=20, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=2, score=0.694782, total=   6.8s
    [CV] max_depth=20, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=2, score=0.701775, total=   7.2s
    [CV] max_depth=20, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=2, score=0.602700, total=   7.2s
    [CV] max_depth=20, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=2, score=0.459522, total=   7.9s
    [CV] max_depth=20, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=5, score=0.587392, total=  10.6s
    [CV] max_depth=20, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=5, score=0.733196, total=  12.5s
    [CV] max_depth=20, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=5, score=0.744406, total=  14.2s
    [CV] max_depth=20, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=5, score=0.650541, total=  12.5s
    [CV] max_depth=20, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=20, max_features=auto, n_estimators=5, score=0.525056, total=  13.7s
    [CV] max_depth=20, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=10, score=0.605057, total=  26.9s
    [CV] max_depth=20, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=10, score=0.748090, total=  26.9s
    [CV] max_depth=20, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=10, score=0.756893, total=  27.5s
    [CV] max_depth=20, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=10, score=0.667678, total=  27.7s
    [CV] max_depth=20, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=10, score=0.545572, total=  27.5s
    [CV] max_depth=20, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=30, score=0.615596, total= 1.1min
    [CV] max_depth=20, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=30, score=0.756136, total= 1.1min
    [CV] max_depth=20, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=30, score=0.765902, total= 1.1min
    [CV] max_depth=20, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=30, score=0.678182, total= 1.1min
    [CV] max_depth=20, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=30, score=0.559657, total= 1.1min
    [CV] max_depth=20, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=50, score=0.618699, total= 1.9min
    [CV] max_depth=20, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=50, score=0.757916, total= 1.9min
    [CV] max_depth=20, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=50, score=0.766704, total=86.7min
    [CV] max_depth=20, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=50, score=0.680446, total= 2.0min
    [CV] max_depth=20, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=20, max_features=auto, n_estimators=50, score=0.560153, total= 1.9min
    [CV] max_depth=20, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=auto, n_estimators=100, score=0.620555, total= 3.6min
    [CV] max_depth=20, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=auto, n_estimators=100, score=0.759156, total= 3.7min
    [CV] max_depth=20, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=auto, n_estimators=100, score=0.769157, total= 3.7min
    [CV] max_depth=20, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=auto, n_estimators=100, score=0.681960, total= 3.7min
    [CV] max_depth=20, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=auto, n_estimators=100, score=0.563258, total= 3.6min
    [CV] max_depth=20, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=1, score=0.426473, total=   3.0s
    [CV] max_depth=20, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=1, score=0.587247, total=   2.9s
    [CV] max_depth=20, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=1, score=0.582284, total=   2.8s
    [CV] max_depth=20, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=1, score=0.497873, total=   2.7s
    [CV] max_depth=20, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=1, score=0.353552, total=   2.4s
    [CV] max_depth=20, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=2, score=0.535385, total=   2.6s
    [CV] max_depth=20, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=2, score=0.689179, total=   2.6s
    [CV] max_depth=20, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=2, score=0.694599, total=   2.6s
    [CV] max_depth=20, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=2, score=0.596650, total=   2.6s
    [CV] max_depth=20, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=2, score=0.471076, total=   2.7s
    [CV] max_depth=20, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=5, score=0.590300, total=   3.6s
    [CV] max_depth=20, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=5, score=0.735235, total=   3.5s
    [CV] max_depth=20, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=5, score=0.748640, total=   3.6s
    [CV] max_depth=20, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=5, score=0.651934, total=   3.7s
    [CV] max_depth=20, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=20, max_features=log2, n_estimators=5, score=0.526550, total=   3.6s
    [CV] max_depth=20, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=10, score=0.614389, total=   7.7s
    [CV] max_depth=20, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=10, score=0.750751, total=   7.9s
    [CV] max_depth=20, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=10, score=0.756200, total=   7.6s
    [CV] max_depth=20, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=10, score=0.669851, total=   7.4s
    [CV] max_depth=20, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=10, score=0.552947, total=   8.2s
    [CV] max_depth=20, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=30, score=0.626634, total=  17.4s
    [CV] max_depth=20, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=30, score=0.757292, total=  19.1s
    [CV] max_depth=20, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=30, score=0.768969, total=  18.6s
    [CV] max_depth=20, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=30, score=0.681579, total=  18.2s
    [CV] max_depth=20, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=30, score=0.564425, total=  18.6s
    [CV] max_depth=20, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=50, score=0.629298, total=  31.2s
    [CV] max_depth=20, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=50, score=0.762244, total=82.5min
    [CV] max_depth=20, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=50, score=0.771996, total=  26.0s
    [CV] max_depth=20, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=50, score=0.683372, total=  32.1s
    [CV] max_depth=20, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=20, max_features=log2, n_estimators=50, score=0.567710, total=  32.1s
    [CV] max_depth=20, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=log2, n_estimators=100, score=0.631978, total= 1.0min
    [CV] max_depth=20, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=log2, n_estimators=100, score=0.763821, total=  59.0s
    [CV] max_depth=20, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=log2, n_estimators=100, score=0.771671, total=  58.7s
    [CV] max_depth=20, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=log2, n_estimators=100, score=0.685984, total=  58.5s
    [CV] max_depth=20, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=20, max_features=log2, n_estimators=100, score=0.569902, total=  58.8s
    [CV] max_depth=50, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=1, score=0.350604, total=  11.4s
    [CV] max_depth=50, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=1, score=0.640720, total=   9.2s
    [CV] max_depth=50, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=1, score=0.665291, total=   8.4s
    [CV] max_depth=50, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=1, score=0.460448, total=   7.1s
    [CV] max_depth=50, max_features=auto, n_estimators=1 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=1, score=0.198864, total=   6.2s
    [CV] max_depth=50, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=2, score=0.495697, total=   6.1s
    [CV] max_depth=50, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=2, score=0.743491, total=   7.1s
    [CV] max_depth=50, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=2, score=0.760303, total=   8.1s
    [CV] max_depth=50, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=2, score=0.593170, total=   8.7s
    [CV] max_depth=50, max_features=auto, n_estimators=2 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=2, score=0.367466, total=   8.6s
    [CV] max_depth=50, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=5, score=0.586766, total=  12.3s
    [CV] max_depth=50, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=5, score=0.801972, total=  13.9s
    [CV] max_depth=50, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=5, score=0.815837, total=  15.3s
    [CV] max_depth=50, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=5, score=0.673046, total=  15.0s
    [CV] max_depth=50, max_features=auto, n_estimators=5 .................
    [CV]  max_depth=50, max_features=auto, n_estimators=5, score=0.460916, total=  15.2s
    [CV] max_depth=50, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=10, score=0.616045, total=  30.3s
    [CV] max_depth=50, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=10, score=0.819622, total=  30.6s
    [CV] max_depth=50, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=10, score=0.834344, total=  30.7s
    [CV] max_depth=50, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=10, score=0.699718, total=  30.6s
    [CV] max_depth=50, max_features=auto, n_estimators=10 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=10, score=0.493935, total=  30.3s
    [CV] max_depth=50, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=30, score=0.637337, total= 1.2min
    [CV] max_depth=50, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=30, score=0.832385, total= 1.3min
    [CV] max_depth=50, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=30, score=0.846693, total= 1.2min
    [CV] max_depth=50, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=30, score=0.716971, total= 1.2min
    [CV] max_depth=50, max_features=auto, n_estimators=30 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=30, score=0.516474, total= 1.2min
    [CV] max_depth=50, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=50, score=0.639542, total= 2.1min
    [CV] max_depth=50, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=50, score=0.835127, total= 2.1min
    [CV] max_depth=50, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=50, score=0.849586, total= 2.1min
    [CV] max_depth=50, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=50, score=0.720089, total= 2.1min
    [CV] max_depth=50, max_features=auto, n_estimators=50 ................
    [CV]  max_depth=50, max_features=auto, n_estimators=50, score=0.520507, total= 2.1min
    [CV] max_depth=50, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=auto, n_estimators=100, score=0.643722, total= 4.1min
    [CV] max_depth=50, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=auto, n_estimators=100, score=0.836893, total= 4.0min
    [CV] max_depth=50, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=auto, n_estimators=100, score=0.851128, total= 4.1min
    [CV] max_depth=50, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=auto, n_estimators=100, score=0.722009, total= 4.1min
    [CV] max_depth=50, max_features=auto, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=auto, n_estimators=100, score=0.525270, total= 4.1min
    [CV] max_depth=50, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=1, score=0.349059, total=   3.4s
    [CV] max_depth=50, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=1, score=0.638589, total=   3.3s
    [CV] max_depth=50, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=1, score=0.665572, total=   3.2s
    [CV] max_depth=50, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=1, score=0.438307, total=   2.7s
    [CV] max_depth=50, max_features=log2, n_estimators=1 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=1, score=0.192254, total=   2.6s
    [CV] max_depth=50, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=2, score=0.503048, total=   2.6s
    [CV] max_depth=50, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=2, score=0.744133, total=   2.9s
    [CV] max_depth=50, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=2, score=0.763344, total=   3.1s
    [CV] max_depth=50, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=2, score=0.592132, total=   3.1s
    [CV] max_depth=50, max_features=log2, n_estimators=2 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=2, score=0.365455, total=   2.8s
    [CV] max_depth=50, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=5, score=0.595623, total=   3.7s
    [CV] max_depth=50, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=5, score=0.806501, total=   3.6s
    [CV] max_depth=50, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=5, score=0.822228, total=   3.8s
    [CV] max_depth=50, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=5, score=0.673854, total=   4.2s
    [CV] max_depth=50, max_features=log2, n_estimators=5 .................
    [CV]  max_depth=50, max_features=log2, n_estimators=5, score=0.453929, total=   4.7s
    [CV] max_depth=50, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=10, score=0.626281, total=   9.2s
    [CV] max_depth=50, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=10, score=0.826891, total=   8.7s
    [CV] max_depth=50, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=10, score=0.840923, total=   9.3s
    [CV] max_depth=50, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=10, score=0.701045, total=   8.6s
    [CV] max_depth=50, max_features=log2, n_estimators=10 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=10, score=0.483902, total=   8.3s
    [CV] max_depth=50, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=30, score=0.647703, total=  21.3s
    [CV] max_depth=50, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=30, score=0.840133, total=  21.7s
    [CV] max_depth=50, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=30, score=0.853625, total=  21.1s
    [CV] max_depth=50, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=30, score=0.716367, total=  20.1s
    [CV] max_depth=50, max_features=log2, n_estimators=30 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=30, score=0.507641, total=  20.3s
    [CV] max_depth=50, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=50, score=0.650888, total=  35.1s
    [CV] max_depth=50, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=50, score=0.842409, total=  35.2s
    [CV] max_depth=50, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=50, score=0.856057, total=  35.1s
    [CV] max_depth=50, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=50, score=0.722225, total=  34.8s
    [CV] max_depth=50, max_features=log2, n_estimators=50 ................
    [CV]  max_depth=50, max_features=log2, n_estimators=50, score=0.512562, total=  34.2s
    [CV] max_depth=50, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=log2, n_estimators=100, score=0.653803, total= 1.1min
    [CV] max_depth=50, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=log2, n_estimators=100, score=0.844444, total= 1.1min
    [CV] max_depth=50, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=log2, n_estimators=100, score=0.858086, total= 1.1min
    [CV] max_depth=50, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=log2, n_estimators=100, score=0.725236, total= 1.1min
    [CV] max_depth=50, max_features=log2, n_estimators=100 ...............
    [CV]  max_depth=50, max_features=log2, n_estimators=100, score=0.513808, total=41.5min
    [CV] max_depth=100, max_features=auto, n_estimators=1 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=1, score=0.342620, total=   3.5s
    [CV] max_depth=100, max_features=auto, n_estimators=1 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=1, score=0.643663, total=   3.8s
    [CV] max_depth=100, max_features=auto, n_estimators=1 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=1, score=0.662976, total=   3.3s
    [CV] max_depth=100, max_features=auto, n_estimators=1 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=1, score=0.468196, total=   3.7s
    [CV] max_depth=100, max_features=auto, n_estimators=1 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=1, score=0.196569, total=   4.0s
    [CV] max_depth=100, max_features=auto, n_estimators=2 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=2, score=0.500745, total=   3.7s
    [CV] max_depth=100, max_features=auto, n_estimators=2 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=2, score=0.744274, total=   3.7s
    [CV] max_depth=100, max_features=auto, n_estimators=2 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=2, score=0.758840, total=   4.0s
    [CV] max_depth=100, max_features=auto, n_estimators=2 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=2, score=0.593082, total=   4.0s
    [CV] max_depth=100, max_features=auto, n_estimators=2 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=2, score=0.359406, total=   3.7s
    [CV] max_depth=100, max_features=auto, n_estimators=5 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=5, score=0.587253, total=   4.7s
    [CV] max_depth=100, max_features=auto, n_estimators=5 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=5, score=0.801032, total=   5.5s
    [CV] max_depth=100, max_features=auto, n_estimators=5 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=5, score=0.815489, total=   5.7s
    [CV] max_depth=100, max_features=auto, n_estimators=5 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=5, score=0.673330, total=   5.4s
    [CV] max_depth=100, max_features=auto, n_estimators=5 ................
    [CV]  max_depth=100, max_features=auto, n_estimators=5, score=0.462788, total=   5.4s
    [CV] max_depth=100, max_features=auto, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=10, score=0.615275, total=  11.7s
    [CV] max_depth=100, max_features=auto, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=10, score=0.819495, total=  11.5s
    [CV] max_depth=100, max_features=auto, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=10, score=0.834740, total=  11.6s
    [CV] max_depth=100, max_features=auto, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=10, score=0.698897, total=  11.6s
    [CV] max_depth=100, max_features=auto, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=10, score=0.495241, total=  11.6s
    [CV] max_depth=100, max_features=auto, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=30, score=0.637472, total=  29.8s
    [CV] max_depth=100, max_features=auto, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=30, score=0.832627, total=  27.1s
    [CV] max_depth=100, max_features=auto, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=30, score=0.846784, total=  30.3s
    [CV] max_depth=100, max_features=auto, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=30, score=0.716362, total=  30.1s
    [CV] max_depth=100, max_features=auto, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=30, score=0.518152, total=  30.0s
    [CV] max_depth=100, max_features=auto, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=50, score=0.640041, total=  51.0s
    [CV] max_depth=100, max_features=auto, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=50, score=0.834798, total=  50.9s
    [CV] max_depth=100, max_features=auto, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=50, score=0.849565, total=  51.5s
    [CV] max_depth=100, max_features=auto, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=50, score=0.719206, total=  49.1s
    [CV] max_depth=100, max_features=auto, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=auto, n_estimators=50, score=0.522479, total=  51.0s
    [CV] max_depth=100, max_features=auto, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=auto, n_estimators=100, score=0.643386, total= 1.7min
    [CV] max_depth=100, max_features=auto, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=auto, n_estimators=100, score=0.836806, total= 1.7min
    [CV] max_depth=100, max_features=auto, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=auto, n_estimators=100, score=0.851108, total=14.8min
    [CV] max_depth=100, max_features=auto, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=auto, n_estimators=100, score=0.722515, total= 1.8min
    [CV] max_depth=100, max_features=auto, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=auto, n_estimators=100, score=0.525046, total= 1.8min
    [CV] max_depth=100, max_features=log2, n_estimators=1 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=1, score=0.357085, total=   1.0s
    [CV] max_depth=100, max_features=log2, n_estimators=1 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=1, score=0.630536, total=   1.1s
    [CV] max_depth=100, max_features=log2, n_estimators=1 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=1, score=0.656754, total=   1.2s
    [CV] max_depth=100, max_features=log2, n_estimators=1 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=1, score=0.463304, total=   1.1s
    [CV] max_depth=100, max_features=log2, n_estimators=1 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=1, score=0.178072, total=   1.0s
    [CV] max_depth=100, max_features=log2, n_estimators=2 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=2, score=0.501769, total=   1.2s
    [CV] max_depth=100, max_features=log2, n_estimators=2 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=2, score=0.742777, total=   1.3s
    [CV] max_depth=100, max_features=log2, n_estimators=2 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=2, score=0.763245, total=   1.3s
    [CV] max_depth=100, max_features=log2, n_estimators=2 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=2, score=0.589270, total=   1.2s
    [CV] max_depth=100, max_features=log2, n_estimators=2 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=2, score=0.353179, total=   1.3s
    [CV] max_depth=100, max_features=log2, n_estimators=5 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=5, score=0.593192, total=   2.0s
    [CV] max_depth=100, max_features=log2, n_estimators=5 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=5, score=0.806747, total=   1.9s
    [CV] max_depth=100, max_features=log2, n_estimators=5 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=5, score=0.820585, total=   2.0s
    [CV] max_depth=100, max_features=log2, n_estimators=5 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=5, score=0.670983, total=   1.9s
    [CV] max_depth=100, max_features=log2, n_estimators=5 ................
    [CV]  max_depth=100, max_features=log2, n_estimators=5, score=0.448761, total=   2.0s
    [CV] max_depth=100, max_features=log2, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=10, score=0.625212, total=   3.8s
    [CV] max_depth=100, max_features=log2, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=10, score=0.825653, total=   4.0s
    [CV] max_depth=100, max_features=log2, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=10, score=0.839518, total=   4.0s
    [CV] max_depth=100, max_features=log2, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=10, score=0.698285, total=   4.4s
    [CV] max_depth=100, max_features=log2, n_estimators=10 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=10, score=0.488069, total=   4.5s
    [CV] max_depth=100, max_features=log2, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=30, score=0.646503, total=  11.9s
    [CV] max_depth=100, max_features=log2, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=30, score=0.839244, total=   9.6s
    [CV] max_depth=100, max_features=log2, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=30, score=0.853031, total=  10.0s
    [CV] max_depth=100, max_features=log2, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=30, score=0.719004, total=  12.4s
    [CV] max_depth=100, max_features=log2, n_estimators=30 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=30, score=0.507412, total=  10.4s
    [CV] max_depth=100, max_features=log2, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=50, score=0.651389, total=  14.3s
    [CV] max_depth=100, max_features=log2, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=50, score=0.842260, total=  16.1s
    [CV] max_depth=100, max_features=log2, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=50, score=0.856257, total=  16.0s
    [CV] max_depth=100, max_features=log2, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=50, score=0.721911, total=  15.8s
    [CV] max_depth=100, max_features=log2, n_estimators=50 ...............
    [CV]  max_depth=100, max_features=log2, n_estimators=50, score=0.511711, total=  15.8s
    [CV] max_depth=100, max_features=log2, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=log2, n_estimators=100, score=0.653762, total=  31.4s
    [CV] max_depth=100, max_features=log2, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=log2, n_estimators=100, score=0.844691, total=  31.7s
    [CV] max_depth=100, max_features=log2, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=log2, n_estimators=100, score=0.857889, total=  33.4s
    [CV] max_depth=100, max_features=log2, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=log2, n_estimators=100, score=0.724079, total=  32.1s
    [CV] max_depth=100, max_features=log2, n_estimators=100 ..............
    [CV]  max_depth=100, max_features=log2, n_estimators=100, score=0.514242, total=  31.3s

    [Parallel(n_jobs=1)]: Done 280 out of 280 | elapsed: 558.9min finished

    Best parameters: {'max_depth': 50, 'max_features': 'log2', 'n_estimators': 100}
    Best score: 0.719075403196
    Best estimators: RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=50,
               max_features='log2', max_leaf_nodes=None,
               min_impurity_split=1e-07, min_samples_leaf=1,
               min_samples_split=2, min_weight_fraction_leaf=0.0,
               n_estimators=100, n_jobs=-1, oob_score=False, random_state=None,
               verbose=0, warm_start=False)

    33574.71210980415

After performing cross validation, here are the results we found:
* **max_depth** = 50
* **max_features** = log2
* **n_estimators** = 100

### Fitting the model
We will use the parameters we found in the previous to fit our random forest model.

```python
reg = RandomForestRegressor(n_estimators=100, max_depth=50, max_features='log2', n_jobs=1)
```

```python
reg.fit(pickup_train, y_pickup_train)
```

    RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=50,
               max_features='log2', max_leaf_nodes=None,
               min_impurity_split=1e-07, min_samples_leaf=1,
               min_samples_split=2, min_weight_fraction_leaf=0.0,
               n_estimators=100, n_jobs=1, oob_score=False, random_state=None,
               verbose=0, warm_start=False)

### Make predictions and evaluating results

```python
predicted_pickups = reg.predict(pickup_test)
accuracy_score(np.round(np.power(10, y_pickup_test)-1, decimals=0),
               np.round(np.power(10, predicted_pickups)-1, decimals=0))
```

    0.43662059133453168

Performing prediction on the 2017 test data, we get only a 43.66% accuracy rate, not so good. However, for our particular study, a low prediction accuracy does not necessary mean that our model is not good. This accuracy score only accounts for the number of pickups that our model has predicted exactly and does not factor in the proximity of our prediction, and considering the scale of the data we are working with, the proximity to the exact solution is pretty important to asses. For example, if a specific location had 1,000 true pickups, but our model predicted that same location would have 998 pickups. That prediction is not exactly correct but since we are working with a dataset containing several million pickups, a prediction that is off by only a factor of 2 is fairly accurate. So to take a further inspection on how good our model is, we must take into account how far are predictions are off from the true values and we can do that by calculating the root mean square error.

#### Calculate RMSE

```python
rmse = np.sqrt(mean_squared_error(predicted_pickups, y_pickup_test))
rmse_back = np.power(10, rmse)
print("RMSE (log-space) = %0.3f" %rmse)
print("RMSE = %0.3f" %rmse_back)
```

    RMSE (log-space) = 0.141
    RMSE = 1.382

Our calculated RMSE for our model was 0.141, but that is in log space so we must transform back. Doing so, we obtain a value of 1.382. That means our That means that the other 56% of our predictions are only a factor of 1.382 or less away from the true value. Now remember we are working with a dataset containing several million pickups, so our model being only a factor of 1.382 pickups away from the true pickup values means that model is predicting pickup values very close to the actual number of pickups. Knowing this, now our 43% prediction accuracy is actually quite considerably good.

#### Most important features

```python
import operator
varImp = dict(zip(list(pickup_train.columns.values), reg.feature_importances_))
varImp_sorted = sorted(varImp.items(), key=operator.itemgetter(1), reverse=True)
varImp_sorted
```

    [('Latitude', 0.27621509508875908),
     ('Longitude', 0.23507218037847466),
     ('Year_num', 0.050127152421569302),
     ('Year_cos', 0.050009352118430349),
     ('Year_sin', 0.049514385038255677),
     ('Day_num', 0.049068821142499879),
     ('Day_cos', 0.0310088031001606),
     ('Month_num', 0.029413920719945082),
     ('Day', 0.029375558051922376),
     ('Month_cos', 0.028045684698258282),
     ('Hour_num', 0.02783126044194419),
     ('Day_sin', 0.027064419463383539),
     ('Month_sin', 0.026630089073298717),
     ('Hour', 0.025994081524676652),
     ('Hour_sin', 0.025958827259884246),
     ('Month', 0.019470573634985684),
     ('Hour_cos', 0.019199795843551602)]

#### Write predicted values to csv file
We will take our predicted pickups and output these values to a csv to create visualizations. We will be using Tableau to visualize our predicted values.

```python
predicted_df = pd.DataFrame({'Latitude': pickup_test['Latitude'],
                             'Longitude': pickup_test['Longitude'],
                             'Year': np.repeat(2017, pickup_test.shape[0]),
                             'Day': pickup_test['Day'],
                             'Hour': pickup_test['Hour'],
                             'Month': pickup_test['Month'],
                             'WDay': x_test['WDay'],
                             'YDay': np.round(pickup_test['Year_num']*365),
                             'Pickups': np.round(np.power(10, predicted_pickups)-1, decimals=0)})
predicted_df.to_csv('PredictedPickups.csv', index=False)
```


```python
from galgebra.printer import Format
from galgebra.ga import Ga
from sympy import symbols
```

```python
Format()
```

```python
coords = (x,y,z) = symbols('x,y,z',real=True)
(o3d,ex,ey,ez) = Ga.build('e_x e_y e_z',g=[1,1,1],coords=coords)
```

```python
v = o3d.mv('v','vector')
```

```python
v
```

\begin{equation*} v = v^{x} \boldsymbol{e}_{x} + v^{y} \boldsymbol{e}_{y} + v^{z} \boldsymbol{e}_{z} \end{equation*}

```python
v.Fmt(3,'v')
```

 \begin{align*}   v =& v^{x} \boldsymbol{e}_{x} \\  &  + v^{y} \boldsymbol{e}_{y} \\  &  + v^{z} \boldsymbol{e}_{z}  \end{align*}

```python
V = o3d.mv('V','vector',f=True)
```

```python
V.Fmt()
```

\begin{equation*} V = V^{x}  \boldsymbol{e}_{x} + V^{y}  \boldsymbol{e}_{y} + V^{z}  \boldsymbol{e}_{z} \end{equation*}

```python
V
```

\begin{equation*} V = V^{x}  \boldsymbol{e}_{x} + V^{y}  \boldsymbol{e}_{y} + V^{z}  \boldsymbol{e}_{z} \end{equation*}

```python
gradV = o3d.grad*V
```

```python
gradV
```

\begin{equation*} \left ( \partial_{x} V^{x}  + \partial_{y} V^{y}  + \partial_{z} V^{z} \right )  + \left ( - \partial_{y} V^{x}  + \partial_{x} V^{y} \right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( - \partial_{z} V^{x}  + \partial_{x} V^{z} \right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( - \partial_{z} V^{y}  + \partial_{y} V^{z} \right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z} \end{equation*}

```python
gradV.Fmt(3,r'\nabla V')
```

 \begin{align*}   \nabla V =& \left ( \partial_{x} V^{x}  + \partial_{y} V^{y}  + \partial_{z} V^{z} \right )  \\  &  + \left ( - \partial_{y} V^{x}  + \partial_{x} V^{y} \right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} \\  &  + \left ( - \partial_{z} V^{x}  + \partial_{x} V^{z} \right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} \\  &  + \left ( - \partial_{z} V^{y}  + \partial_{y} V^{z} \right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}  \end{align*}

```python
gradV.Fmt(2,r'\nabla V')
```

 \begin{align*}   \nabla V =& \left ( \partial_{x} V^{x}  + \partial_{y} V^{y}  + \partial_{z} V^{z} \right )  \\  &  + \left ( - \partial_{y} V^{x}  + \partial_{x} V^{y} \right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( - \partial_{z} V^{x}  + \partial_{x} V^{z} \right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( - \partial_{z} V^{y}  + \partial_{y} V^{z} \right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}  \end{align*}

```python
lap = o3d.grad|o3d.grad
```

```python
lap
```

\begin{equation*} \frac{\partial^{2}}{\partial x^{2}} + \frac{\partial^{2}}{\partial y^{2}} + \frac{\partial^{2}}{\partial z^{2}} \end{equation*}

```python
lap.Fmt(1,'\\nabla^{2}')
```

\begin{equation*} \nabla^{2} = \frac{\partial^{2}}{\partial x^{2}} + \frac{\partial^{2}}{\partial y^{2}} + \frac{\partial^{2}}{\partial z^{2}} \end{equation*}

```python
A = o3d.lt('A')
```

```python
A
```

\begin{equation*} \left \{ \begin{array}{ll} L \left ( \boldsymbol{e}_{x}\right ) =& A_{xx} \boldsymbol{e}_{x} + A_{yx} \boldsymbol{e}_{y} + A_{zx} \boldsymbol{e}_{z} \\ L \left ( \boldsymbol{e}_{y}\right ) =& A_{xy} \boldsymbol{e}_{x} + A_{yy} \boldsymbol{e}_{y} + A_{zy} \boldsymbol{e}_{z} \\ L \left ( \boldsymbol{e}_{z}\right ) =& A_{xz} \boldsymbol{e}_{x} + A_{yz} \boldsymbol{e}_{y} + A_{zz} \boldsymbol{e}_{z}  \end{array} \right \}
 \end{equation*}

```python
A.Fmt(1,'A')
```

\begin{equation*} \left \{ \begin{array}{ll} L \left ( \boldsymbol{e}_{x}\right ) =& A_{xx} \boldsymbol{e}_{x} + A_{yx} \boldsymbol{e}_{y} + A_{zx} \boldsymbol{e}_{z} \\ L \left ( \boldsymbol{e}_{y}\right ) =& A_{xy} \boldsymbol{e}_{x} + A_{yy} \boldsymbol{e}_{y} + A_{zy} \boldsymbol{e}_{z} \\ L \left ( \boldsymbol{e}_{z}\right ) =& A_{xz} \boldsymbol{e}_{x} + A_{yz} \boldsymbol{e}_{y} + A_{zz} \boldsymbol{e}_{z}  \end{array} \right \}
 \end{equation*}

```python
A.Fmt(2,'A')
```

\begin{equation*} \left \{ \begin{array}{ll} L \left ( \boldsymbol{e}_{x}\right ) =& A_{xx} \boldsymbol{e}_{x} + A_{yx} \boldsymbol{e}_{y} + A_{zx} \boldsymbol{e}_{z} \\ L \left ( \boldsymbol{e}_{y}\right ) =& A_{xy} \boldsymbol{e}_{x} + A_{yy} \boldsymbol{e}_{y} + A_{zy} \boldsymbol{e}_{z} \\ L \left ( \boldsymbol{e}_{z}\right ) =& A_{xz} \boldsymbol{e}_{x} + A_{yz} \boldsymbol{e}_{y} + A_{zz} \boldsymbol{e}_{z}  \end{array} \right \}
 \end{equation*}

```python

```


# 02 - Reverse Time Migration

This notebook is the second in a series of tutorial highlighting various aspects of seismic inversion based on Devito operators. In this second example we aim to highlight the core ideas behind seismic inversion, where we create an image of the subsurface from field recorded data. This tutorial follows on the modelling tutorial and will reuse the modelling operator and velocity model.

## Imaging requirement

Seismic imaging relies on two known parameters:

- **Field data** - or also called **recorded data**. This is a shot record corresponding to the true velocity model. In practice this data is acquired as described in the first tutorial. In order to simplify this tutorial we will generate synthetic field data by modelling it with the **true velocity model**.

- **Background velocity model**. This is a velocity model that has been obtained by processing and inverting the field data. We will look at this methods in the following tutorial as it relies on the method we are describing here. This velocity model is usually a **smooth version** of the true velocity model.

## Imaging computational setup

In this tutorial, we will introduce the back-propagation operator. This operator simulates the adjoint wave-equation, that is a wave-equation solved in a reversed time order. This time reversal led to the naming of the method we present here, called Reverse Time Migration. The notion of adjoint in exploration geophysics is fundamental as most of the wave-equation based imaging and inversion methods rely on adjoint based optimization methods.

## Notes on the operators

As we have already described the creation of a forward modelling operator, we will use a thin wrapper function instead. This wrapper is provided by a utility class called `AcousticWaveSolver`, which provides all the necessary operators for seismic modeling, imaging and inversion. The `AcousticWaveSolver` provides a more concise API for common wave propagation operators and caches the Devito `Operator` objects to avoid unnecessary recompilation. Operators introduced for the first time in this tutorial will be properly described.

As before we initialize printing and import some utilities. We also raise the Devito log level to avoid excessive logging for repeated operator invocations.

```python
import numpy as np
%matplotlib inline

from devito import configuration
configuration['log-level'] = 'WARNING'
```

## Computational considerations

Seismic inversion algorithms are generally very computationally demanding and require a large amount of memory to store the forward wavefield. In order to keep this tutorial as lightweight as possible we are using a very simple
velocity model that requires low temporal and spatial resolution. For a more realistic model, a second set of preset parameters for a reduced version of the 2D Marmousi data set [1] is provided below in comments. This can be run to create some more realistic subsurface images. However, this second preset is more computationally demanding and requires a slightly more powerful workstation.

```python
# Configure model presets
from examples.seismic import demo_model

# Enable model presets here:
preset = 'layers-isotropic'  # A simple but cheap model (recommended)
# preset = 'marmousi2d-isotropic'  # A larger more realistic model

# Standard preset with a simple two-layer model
if preset == 'layers-isotropic':
    def create_model(grid=None):
        return demo_model('layers-isotropic', origin=(0., 0.), shape=(101, 101),
                          spacing=(10., 10.), nbl=20, grid=grid, nlayers=2)
    filter_sigma = (1, 1)
    nshots = 21
    nreceivers = 101
    t0 = 0.
    tn = 1000.  # Simulation last 1 second (1000 ms)
    f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)

# A more computationally demanding preset based on the 2D Marmousi model
if preset == 'marmousi2d-isotropic':
    def create_model(grid=None):
        return demo_model('marmousi2d-isotropic', data_path='../../../../data/',
                          grid=grid, nbl=20)
    filter_sigma = (6, 6)
    nshots = 301  # Need good covergae in shots, one every two grid points
    nreceivers = 601  # One recevier every grid point
    t0 = 0.
    tn = 3500.  # Simulation last 3.5 second (3500 ms)
    f0 = 0.025  # Source peak frequency is 25Hz (0.025 kHz)
```

# True and smooth velocity models

First, we create the model data for the "true" model from a given demonstration preset. This model represents the subsurface topology for the purposes of this example and we will later use it to generate our synthetic data readings. We also generate a second model and apply a smoothing filter to it, which represents our initial model for the imaging algorithm. The perturbation between these two models can be thought of as the image we are trying to recover.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_velocity, plot_perturbation
from devito import gaussian_smooth

# Create true model from a preset
model = create_model()

# Create initial model and smooth the boundaries
model0 = create_model(grid=model.grid)
gaussian_smooth(model0.vp, sigma=filter_sigma)

# Plot the true and initial model and the perturbation between them
plot_velocity(model)
plot_velocity(model0)
plot_perturbation(model0, model)
```

## Acquisition geometry

Next we define the positioning and the wave signal of our source, as well as the location of our receivers. To generate the wavelet for our source we require the discretized values of time that we are going to use to model a single "shot",
which again depends on the grid spacing used in our model. For consistency this initial setup will look exactly as in the previous modelling tutorial, although we will vary the position of our source later on during the actual imaging algorithm.

```python
#NBVAL_IGNORE_OUTPUT
# Define acquisition geometry: source
from examples.seismic import AcquisitionGeometry

# First, position source centrally in all dimensions, then set depth
src_coordinates = np.empty((1, 2))
src_coordinates[0, :] = np.array(model.domain_size) * .5
src_coordinates[0, -1] = 20.  # Depth is 20m

# Define acquisition geometry: receivers

# Initialize receivers for synthetic and imaging data
rec_coordinates = np.empty((nreceivers, 2))
rec_coordinates[:, 0] = np.linspace(0, model.domain_size[0], num=nreceivers)
rec_coordinates[:, 1] = 30.

# Geometry

geometry = AcquisitionGeometry(model, rec_coordinates, src_coordinates, t0, tn, f0=.010, src_type='Ricker')
# We can plot the time signature to see the wavelet
geometry.src.show()
```

# True and smooth data

We can now generate the shot record (receiver readings) corresponding to our true and initial models. The difference between these two records will be the basis of the imaging procedure.

For this purpose we will use the same forward modelling operator that was introduced in the previous tutorial, provided by the `AcousticWaveSolver` utility class. This object instantiates a set of pre-defined operators according to an initial definition of the acquisition geometry, consisting of source and receiver symbols. The solver objects caches the individual operators and provides a slightly more high-level API that allows us to invoke the modelling modelling operators from the initial tutorial in a single line. In the following cells we use this to generate shot data by only specifying the respective model symbol `m` to use, and the solver will create and return a new `Receiver` object the represents the readings at the previously defined receiver coordinates.

```python
# Compute synthetic data with forward operator
from examples.seismic.acoustic import AcousticWaveSolver

solver = AcousticWaveSolver(model, geometry, space_order=4)
true_d , _, _ = solver.forward(vp=model.vp)
```

```python
# Compute initial data with forward operator
smooth_d, _, _ = solver.forward(vp=model0.vp)
```

```python
#NBVAL_IGNORE_OUTPUT
# Plot shot record for true and smooth velocity model and the difference
from examples.seismic import plot_shotrecord

plot_shotrecord(true_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data - true_d.data, model, t0, tn)
```

# Imaging with back-propagation

As explained in the introduction of this tutorial, this method is based on back-propagation.

## Adjoint wave equation

If we go back to the modelling part, we can rewrite the simulation as a linear system solve:

\begin{equation}
\mathbf{A}(\mathbf{m}) \mathbf{u} = \mathbf{q}
\end{equation}

where $\mathbf{m}$ is the discretized square slowness, $\mathbf{q}$ is the discretized source and $\mathbf{A}(\mathbf{m})$ is the discretized wave-equation. The discretized wave-equation matricial representation is a lower triangular matrix that can be solve with forward substitution. The pointwise writing or the forward substitution leads to the time-stepping stencil.

On a small problem one could form the matrix explicitly and transpose it to obtain the adjoint discrete wave-equation:

\begin{equation}
\mathbf{A}(\mathbf{m})^T \mathbf{v} = \delta \mathbf{d}
\end{equation}

where $\mathbf{v}$ is the discrete **adjoint wavefield** and  $\delta \mathbf{d}$ is the data residual defined as the difference between the field/observed data and the synthetic data $\mathbf{d}_s = \mathbf{P}_r \mathbf{u}$. In our case we derive the discrete adjoint wave-equation from the discrete forward wave-equation to get its stencil.

## Imaging

Wave-equation based imaging relies on one simple concept:

- If the background velocity model is cinematically correct, the forward wavefield $\mathbf{u}$ and the adjoint wavefield $\mathbf{v}$ meet at the reflectors position at zero time offset.

The sum over time of the zero time-offset correlation of these two fields then creates an image of the subsurface. Mathematically this leads to the simple imaging condition:

\begin{equation}
  \text{Image} = \sum_{t=1}^{n_t} \mathbf{u}[t] \mathbf{v}[t]
\end{equation}

In the following tutorials we will describe a more advanced imaging condition that produces shaper and more accurate results.

## Operator

We will now define the imaging operator that computes the adjoint wavefield $\mathbf{v}$ and correlates it with the forward wavefield $\mathbf{u}$. This operator essentially consists of three components:
* Stencil update of the adjoint wavefield `v`
* Injection of the data residual at the adjoint source (forward receiver) location
* Correlation of `u` and `v` to compute the image contribution at each timestep

```python
# Define gradient operator for imaging
from devito import TimeFunction, Operator, Eq, solve
from examples.seismic import PointSource

def ImagingOperator(model, image):
    # Define the wavefield with the size of the model and the time dimension
    v = TimeFunction(name='v', grid=model.grid, time_order=2, space_order=4)

    u = TimeFunction(name='u', grid=model.grid, time_order=2, space_order=4,
                     save=geometry.nt)

    # Define the wave equation, but with a negated damping term
    eqn = model.m * v.dt2 - v.laplace + model.damp * v.dt.T

    # Use `solve` to rearrange the equation into a stencil expression
    stencil = Eq(v.backward, solve(eqn, v.backward))

    # Define residual injection at the location of the forward receivers
    dt = model.critical_dt
    residual = PointSource(name='residual', grid=model.grid,
                           time_range=geometry.time_axis,
                           coordinates=geometry.rec_positions)
    res_term = residual.inject(field=v.backward, expr=residual * dt**2 / model.m)

    # Correlate u and v for the current time step and add it to the image
    image_update = Eq(image, image - u * v)

    return Operator([stencil] + res_term + [image_update],
                    subs=model.spacing_map)
```

## Implementation of the imaging loop

As just explained, the forward wave-equation is solved forward in time while the adjoint wave-equation is solved in a reversed time order. Therefore, the correlation of these two fields over time requires to store one of the two fields. The computational procedure for imaging follows:

- Simulate the forward wave-equation with the background velocity model to get the synthetic data and save the full wavefield $\mathbf{u}$
- Compute the data residual
- Back-propagate the data residual and compute on the fly the image contribution at each time step.

This procedure is applied to multiple source positions (shots) and summed to obtain the full image of the subsurface. We can first visualize the varying locations of the sources that we will use.

```python
#NBVAL_IGNORE_OUTPUT

# Prepare the varying source locations
source_locations = np.empty((nshots, 2), dtype=np.float32)
source_locations[:, 0] = np.linspace(0., 1000, num=nshots)
source_locations[:, 1] = 30.

plot_velocity(model, source=source_locations)
```

```python
# Run imaging loop over shots
from devito import Function

# Create image symbol and instantiate the previously defined imaging operator
image = Function(name='image', grid=model.grid)
op_imaging = ImagingOperator(model, image)

for i in range(nshots):
    print('Imaging source %d out of %d' % (i+1, nshots))

    # Update source location
    geometry.src_positions[0, :] = source_locations[i, :]

    # Generate synthetic data from true model
    true_d, _, _ = solver.forward(vp=model.vp)

    # Compute smooth data and full forward wavefield u0
    smooth_d, u0, _ = solver.forward(vp=model0.vp, save=True)

    # Compute gradient from the data residual
    v = TimeFunction(name='v', grid=model.grid, time_order=2, space_order=4)
    residual = smooth_d.data - true_d.data
    op_imaging(u=u0, v=v, vp=model0.vp, dt=model0.critical_dt,
               residual=residual)

```

    Imaging source 1 out of 21
    Imaging source 2 out of 21
    Imaging source 3 out of 21
    Imaging source 4 out of 21
    Imaging source 5 out of 21
    Imaging source 6 out of 21
    Imaging source 7 out of 21
    Imaging source 8 out of 21
    Imaging source 9 out of 21
    Imaging source 10 out of 21
    Imaging source 11 out of 21
    Imaging source 12 out of 21
    Imaging source 13 out of 21
    Imaging source 14 out of 21
    Imaging source 15 out of 21
    Imaging source 16 out of 21
    Imaging source 17 out of 21
    Imaging source 18 out of 21
    Imaging source 19 out of 21
    Imaging source 20 out of 21
    Imaging source 21 out of 21

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_image

# Plot the inverted image
plot_image(np.diff(image.data, axis=1))
```

```python
from devito import norm
assert np.isclose(norm(image), 1e7, rtol=1e1)
```

And we have an image of the subsurface with a strong reflector at the original location.

## References

[1] _Versteeg, R.J. & Grau, G. (eds.) (1991): The Marmousi experience. Proc. EAGE workshop on Practical Aspects of Seismic Data Inversion (Copenhagen, 1990), Eur. Assoc. Explor. Geophysicists, Zeist._


```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as st
import warnings
warnings.filterwarnings("ignore")
```

* Please email me (ianzhang@connect.hku.hk) if you find any mistakes in the script.

* My consultation timeslot: Wednesday, 16:00 - 17:00, RR113.

# Data Augmentation

Given the model
$$p(\theta,z|\mathbf{y}_{\mathrm{obs}}),$$
we can obtain the **complete-data posterior distribution**
$$p(\theta|\mathbf{y}_{\mathrm{obs}}, z),$$
and the **conditional predictive distribution**
$$p(z|\mathbf{y}_{\mathrm{obs}}, \theta).$$
Our goal is to draw samples from $p(\theta|\mathbf{y}_{\mathrm{obs}})$, where
$$
p(\theta|\mathbf{y}_{\mathrm{obs}})=\int p(\theta,z|\mathbf{y}_{\mathrm{obs}})dz.
$$

## The original DA algorithm (multiple imputation)

I-step: Draw samples $\{\theta^{(k,j)}\}_{j=1}^m$ from $p_k(\theta|\mathbf{y}_{\mathrm{obs}})$, and obtain $\{z^{(k,j)}\}_{j=1}^m$, where $z^{(k,j)}\sim p(z|\mathbf{y}_{\mathrm{obs}},\theta^{(k,j)})$.

P-step: Update the posterior:
$$
p_{k+1}(\theta|\mathbf{y}_{\mathrm{obs}})=\frac{1}{m}\sum_{j=1}^mp(\theta|\mathbf{y}_{\mathrm{obs}},z^{(k,j)}).
$$

## The DA algorithm

Some insights about the original DA algorithm:
* The P-step can be integrated into the I-step.
* The I-step can be modified in the following way:

(1) Draw samples $\{\theta^{(k,j)}\}_{j=1}^m$ from $p_k(\theta|\mathbf{y}_{\mathrm{obs}})$, where
$$
p_k(\theta|\mathbf{y}_{\mathrm{obs}})=\frac{1}{m}\sum_{j=1}^mp(\theta|\mathbf{y}_{\mathrm{obs}},z^{(k-1,j)}).
$$
(2) Draw samples $\{z^{(k,j)}\}_{j=1}^m$ from $p_k(z|\mathbf{y}_{\mathrm{obs}})$, where
$$
p_k(z|\mathbf{y}_{\mathrm{obs}})=\frac{1}{m}\sum_{j=1}^mp(z|\mathbf{y}_{\mathrm{obs}},\theta^{(k,j)}).
$$
But the step (2) is based on **stratification**.

What if we let the step (1) to be based on stratification too? In that case we would have

(1) Draw samples $\{\theta^{(k,j)}\}_{j=1}^m$, where $\theta^{(k,j)}\sim p(\theta|\mathbf{y}_{\mathrm{obs}},z^{(k-1,j)})$.

(2) Draw samples $\{z^{(k,j)}\}_{j=1}^m$, where $z^{(k,j)}\sim p(z|\mathbf{y}_{\mathrm{obs}},\theta^{(k,j)})$.

And we find that this is equivalent to do $m$ **parallel Gibbs**!

So why not just simplify it to be one single Gibbs:

(1) Draw sample $\theta^{(k)}$, where $\theta^{(k)}\sim p(\theta|\mathbf{y}_{\mathrm{obs}},z^{(k-1)})$.

(2) Draw sample $z^{(k)}$, where $z^{(k)}\sim p(z|\mathbf{y}_{\mathrm{obs}},\theta^{(k)})$.

This is the **standard DA algrithm**. We can also draw $z$ first, just choose the convenient one.

## Example: grouped count data (missing data)

Given the observed data (\# of passages, \# of observations): \{(0,139),(1,128),(2,55),(3,25),(4+,13)\}

Assume the **Poisson model**:
$$
P(X=x)=\frac{e^{-\lambda}\lambda^x}{x!},\; x=0,1,\dots
$$
for the \# of passages. Try to estimate $\lambda$.

There are $347=139+128+55+25$ observations $\mathbf{x}_{\mathrm{obs}}$ and 13 missing data $z_{1:13}$.

If we assume the prior $p(\lambda)\propto 1/\lambda$, the model is
$$
p(\lambda,z_{1:13}|\mathbf{x}_{\mathrm{obs}})\propto\frac{1}{\lambda}e^{-347\lambda}\lambda^{128+55\times2+25\times3}\prod_{j=1}^{13}\frac{e^{-\lambda}\lambda^{z_j}}{z_j!}\mathbf{1}(z_j\ge4).
$$
so, we can obtain the complete-data posterior distribution
$$\lambda|\mathbf{x}_{\mathrm{obs}}, z_{1:13}\sim Gamma(313+\sum_{j=1}^{13}z_j,360),$$
and the conditional predictive distribution
$$p(z_j|\mathbf{y}_{\mathrm{obs}}, \lambda)=p(z_j|\lambda)\propto\frac{e^{-\lambda}\lambda^{z_j}}{z_j!}\mathbf{1}(z_j\ge4).$$

Based on the Gibbs, we can obtain the samples $\{\lambda^{(i)},z_{1:13}^{(i)}\}_{i=1}^{n}$.

* Assume $\lambda^{(0)}=\frac{313+13\times4}{360}$. Compare the two estimators of $\lambda$: (1) $\hat{\lambda}_1=\frac{1}{n}\sum_{i=1}^n\lambda^{(i)}$; (2) $\hat{\lambda}_2=\frac{1}{n}\sum_{i=1}^n\frac{313+\sum_{j=1}^{13}z_j^{(i)}}{360}$.

```python

```

## Example: probit model (latent variables)

Other than the logistic model, another classic model in binary classification is called the **probit model**:
$$
P(Y=1|x)=\Phi(x^T\beta),
$$
where $Y$ is a binary label, x is the vector of features, $\beta$ is the model parameter and $\Phi$ is the cdf of the standard normal distribution.

Given data $\{(y_1,x_1),\dots,(y_m,x_m)\}$, and the non-informative prior $p(\beta)\propto 1$, the model is
$$
p(\beta|y_{1:m},x_{1:m})\propto p(y_{1:m},\beta|x_{1:m})=p(y_{1:m}|\beta,x_{1:m})p(\beta)=\prod_{j=1}^m\Phi(x_j^T\beta)^{y_j}(1-\Phi(x_j^T\beta))^{1-y_j},
$$
and we can apply the Random Walk to do inference about $\beta$.

Another way is to use the data augmentation with the Gibbs sampler. To achieve this, we assume the latent variables $z_j=x_j^T\beta+\varepsilon_j$, $j=1,\dots,m$, where $\varepsilon_j\sim N(0,1)$. And we assume $y_j$ depends on $z_j$ deterministically that $y_j=\mathbf{1}(z_j>0)$. To see why this assumption holds, just notice that
$$
P(Y=1|x)=P(\mathbf{1}(Z>0)=1|x)=P(x^T\beta+\varepsilon>0)=P(-\varepsilon<x^T\beta)=\Phi(x^T\beta).
$$
So, the augmented model is
\begin{align}
p(\beta,z_{1:m}|y_{1:m},x_{1:m})&\propto p(y_{1:m},z_{1:m},\beta|x_{1:m})=p(y_{1:m}|z_{1:m})p(z_{1:m}|\beta,x_{1:m})p(\beta)\\
&=\prod_{j=1}^m(\mathbf{1}(z_j>0)\mathbf{1}(y_j=1)+\mathbf{1}(z_j\le0)\mathbf{1}(y_j=0))\exp(-\frac{(z_j-x_j^T\beta)^2}{2}).
\end{align}
Easy to derive that
$$
\beta|z_{1:m},y_{1:m},x_{1:m}\sim N((X^TX)^{-1}X^Tz_{1:m},(X^TX)^{-1}),
$$
where $X=[x_{1:m}]_{m\times p}$, and
$$
p(z_j|\beta,y_{1:m},x_{1:m})\propto(\mathbf{1}(z_j>0)\mathbf{1}(y_j=1)+\mathbf{1}(z_j\le0)\mathbf{1}(y_j=0))\exp(-\frac{(z_j-x_j^T\beta)^2}{2}).
$$

Based on the Gibbs, we can obtain the samples $\{\beta^{(i)},z_{1:m}^{(i)}\}_{i=1}^{n}$.

* Draw $x_j\sim N(\mathbf{0},I_2)$, $j=1,\dots,20$, and draw $y_{1:20}$ from the probit model with $\beta_0=[-1,2]^T$. Try to draw samples of $\beta$ given $x_{1:20},y_{1:20}$. Compare the trajectories of the Random Walk and the Gibbs. Which one is better?

```python

```

```python

```

# Simulating Annealing

For any distribution $\pi(x)$, its negative log function is called the corresponding **energy function**:
$$
h(x)=-\log(\pi(x)).
$$
In the simulated annealing, based on some intuitions from physics, we redefine the distribution with a new parameter $T$ - **the temperature**:
$$
\pi_T(x)=\exp(-\frac{h(x)}{T})(=\pi(x)^{1/T}).
$$

An example:

```python
pi = lambda x: 0.5 * st.norm(loc=-2, scale=0.2).pdf(x) + 0.5 * st.norm(loc=1, scale=0.4).pdf(x)
h = lambda x: -np.log(pi(x))

grid_x = np.linspace(-4, 4, 1000)
fig, ax = plt.subplots(2, 1, figsize=[8, 5])
ax[0].plot(grid_x, pi(grid_x))
ax[1].plot(grid_x, h(grid_x))
plt.show()
```

What if $T$ goes to 0 or infinity?

```python
piT = lambda x, T: np.exp(-h(x) / T)
hight = piT(-2, T=1)

grid_x = np.linspace(-4, 4, 1000)
fig, ax = plt.subplots(figsize=[8, 4])
ax.plot(grid_x, piT(grid_x, T=0.001) * hight / piT(-2, T=0.001), label='T=0.001')
ax.plot(grid_x, piT(grid_x, T=0.1) * hight / piT(-2, T=0.1), label='T=0.1')
ax.plot(grid_x, piT(grid_x, T=1), label='T=1')
ax.plot(grid_x, piT(grid_x, T=10) * hight / piT(-2, T=10), label='T=10')
ax.plot(grid_x, piT(grid_x, T=1000) * hight / piT(-2, T=1000), label='T=1000')
ax.legend()
plt.show()
```

The idea of **Simulating Annealing** is simply to run the **Random Walk** against a redefined target distribution $\pi_T(x)$, and gradually reduce $T$ as the algorithm proceeds. As the temperature decreases, the region around the maximum point would have higher and higher probability to capture the last sample point.

* Run the Random Walk against the above distribution with Gaussian kernel and step size 2, and set $T=2/n$, where $n$ is iteration number. Begin the walk at the local maximum $x_0=1$ and the total iteration time is $N=200$. What's the probability that the last sample point is trapped by the area around the global maximum point $x=-2$. Run 100 repetitions to estimate this probability and draw all the trajectories of the Random Walk.

```python

```

**Think further:** Try different parameter settings to see how different parameters influence the annealing.

```python

```

# Root Finding

A common root ﬁnding problem in statistics: assume
$$
l(\theta)=\log(p(y_{\mathrm{obs}}|\theta)),
$$
and solve $f(\theta)=l'(\theta)=0$ to obtain MLE.

## Bisection method

The assumptions of the **Bisection method** are
1. $f(\theta)$ is one dimensional;
2. $f(\theta)$ is continuous;
3. We have initial points $a, b$ such that $f(a)f(b)<0$.

## Convex and concave functions

```python
x = np.linspace(-1, 1, 1000)
fv = lambda x: np.abs(x)
fc = lambda x: - x**2

fig, ax = plt.subplots(1, 2, figsize=[12, 4])
ax[0].plot(x, fv(x))
ax[0].set_title('convex')
ax[1].plot(x, fc(x))
ax[1].set_title('concave')
plt.show()
```

## Newton–Raphson method

Based on the **Taylor expansion**, we have
$$
f(\theta)=f(\theta_0)+f'(\theta_0)(\theta-\theta_0)+\mathcal{o}(\theta-\theta_0).
$$
If we ignore $\mathcal{o}(\theta-\theta_0)$, $f(\theta_1)=0$ induce
$$
\theta_1=\theta_0-\frac{f(\theta_0)}{f'(\theta_0)}.
$$
This makes an iteration of the **Newton–Raphson algorithm**.

### Multiple dimensions

If $\theta$ is $d$-dimensional, $f(\theta)=\nabla l(\theta)$ would be a multivalued function on a $d$-dimensional space, but we still have the Taylor expansion
$$
\nabla l(\theta)=\nabla l(\theta_0)+\nabla^2 l(\theta_0)(\theta-\theta_0)+\mathcal{o}(||\theta-\theta_0||),
$$
which results in an iteration of the Newton–Raphson algorithm in the $d$-dimensional space
$$
\theta_1=\theta_0+I(\theta_0)^{-1}\nabla l(\theta_0),
$$
where $I(\theta_0)=-\nabla^2 l(\theta_0)$ is called the **observed information matrix**. Different from the Bisection method, the Newton–Raphson algorithm can be easily generalized to high dimension space.

To speed up the algrithm, a simple idea called the **Fisher scoring algorithm** replaces $I(\theta_0)$ by $J(\theta_0)=\mathbb{E}[I(\theta_0)]$ which is called the **Fisher/expected information matrix**. This replacement is valid because
\begin{align}
\frac{1}{n}I(\theta_0)&=-\frac{1}{n}\nabla^2 l(\theta_0)=-\frac{1}{n}\nabla^2 \log(p(y_{\mathrm{obs}}|\theta_0))=-\frac{1}{n}\nabla^2 \log(\prod_{i=1}^np(y_i|\theta_0))=-\frac{1}{n}\sum_{i=1}^n\nabla^2\log(p(y_i|\theta_0))\\
&\approx-\mathbb{E}[\nabla^2\log(p(y_i|\theta_0))]=\frac{1}{n}\mathbb{E}[I(\theta_0)]=\frac{1}{n}J(\theta_0),
\end{align}
and
$$
I(\theta_0)^{-1}\nabla l(\theta_0)=(\frac{1}{n}I(\theta_0))^{-1}\frac{1}{n}\nabla l(\theta_0)\approx(\frac{1}{n}J(\theta_0))^{-1}\frac{1}{n}\nabla l(\theta_0)=J(\theta_0)^{-1}\nabla l(\theta_0).
$$

```python

```

STAT6011 Computational Statistics/Tut 9/Tut 9 (only questions).ipynb