# Online Drift Detection on the Wine Quality Dataset

In the context of deployed models, data (model queries) usually arrive sequentially and we wish to detect it as soon as possible after its occurence. One approach is to perform a test for drift every $W$ time-steps, using the $W$ samples that have arrived since the last test. Such a strategy could be implemented using any of the offline detectors implemented in `alibi-detect`, but being both sensitive to slight drift and responsive to severe drift is difficult. If the window size $W$ is too small then slight drift will be undetectable. If it is too large then the delay between test-points hampers responsiveness to severe drift.

An alternative strategy is to perform a test each time data arrives. However the usual offline methods are not applicable because the process for computing p-values is too expensive and doesn't account for correlated test outcomes when using overlapping windows of test data.

Online detectors instead work by computing the test-statistic once using the first $W$ data points and then updating the test-statistic sequentially at low cost. When no drift has occured the test-statistic fluctuates around its expected value and once drift occurs the test-statistic starts to drift upwards. When it exceeds some preconfigured threshold value, drift is detected.

Unlike offline detectors which require the specification of a threshold p-value (a false positive rate), the online detectors in `alibi-detect` require the specification of an expected run-time (ERT) (an inverted FPR). This is the number of time-steps that we insist our detectors, on average, should run for in the absense of drift before making a false detection. Usually we would like the ERT to be large, however this results in insensitive detectors which are slow to respond when drift does occur. There is a tradeoff between the expected run time and the expected detection delay.

To target the desired ERT, thresholds are configured during an initial configuration phase via simulation. This configuration process is only suitable when the amount reference data (most likely the training data of the model of interest) is relatively large (ideally around an order of magnitude larger than the desired ERT). Configuration can be expensive (less so with a GPU) but allows the detector to operate at low-cost during deployment.

This notebook demonstrates online drift detection using two different two-sample distance metrics for the test-statistic, the maximum mean discrepency (MMD) and least-squared density difference (LSDD), both of which can be updated sequentially at low cost.

### Backend

The online detectors are implemented in both the *PyTorch* and *TensorFlow* frameworks with support for CPU and GPU. Various preprocessing steps are also supported out-of-the box in Alibi Detect for both frameworks and an example will be given in this notebook. Alibi Detect does however not install PyTorch for you. Check the [PyTorch docs](https://pytorch.org/) how to do this.

### Dataset

The [Wine Quality Data Set](https://archive.ics.uci.edu/ml/datasets/wine+quality) consists of 4898 and 1599 samples of white and red wine respectively. Each sample has an associated quality (as determined by experts) and 11 numeric features indicating its acidity, density, pH etc. We consider the regression problem of tring to predict the quality of white wine samples given these features. We will then consider whether the model remains suitable for predicting the quality of red wine samples or whether the associated change in the underlying distribution should be considered as drift.

## Online detection with MMD and Pytorch

The Maximum Mean Discepency (MMD) is a distance-based measure between 2 distributions *p* and *q* based on the mean embeddings $\mu_{p}$ and $\mu_{q}$ in a reproducing kernel Hilbert space $F$:

\begin{align}
MMD(F, p, q) & = || \mu_{p} - \mu_{q} ||^2_{F} \\
\end{align}

Given reference samples $\{X_i\}_{i=1}^{N}$ and test samples $\{Y_i\}_{i=t}^{t+W}$ we may compute an unbiased estimate $\widehat{MMD}^2(F, \{X_i\}_{i=1}^N, \{Y_i\}_{i=t}^{t+W})$ of the squared MMD between the two underlying distributions. Depending on the size of the reference and test windows, $N$ and $W$ respectively, this can be relatively expensive. However, once computed it is possible to update the statistic to estimate to the squared MMD between the distributions underlying $\{X_i\}_{i=1}^{N}$ and $\{Y_i\}_{i=t+1}^{t+1+W}$ at a very low cost, making it suitable for online drift detection.

By default we use a [radial basis function kernel](https://en.wikipedia.org/wiki/Radial_basis_function_kernel), but users are free to pass their own kernel of preference to the detector.

```python
from functools import partial
import matplotlib.pyplot as plt
import numpy as np
import torch
import tensorflow as tf
from torch import nn
import pandas as pd
import scipy

from alibi_detect.models.pytorch.trainer import trainer
from alibi_detect.cd.utils import encompass_batching

np.random.seed(0)
torch.manual_seed(0)
tf.random.set_seed(0)
```

### Load data

First we load in the data:

```python
red = pd.read_csv(
    "https://storage.googleapis.com/seldon-datasets/wine_quality/winequality-red.csv", sep=';'
)
white = pd.read_csv(
    "https://storage.googleapis.com/seldon-datasets/wine_quality/winequality-white.csv", sep=';'
)
white.describe()
```

      fixed acidity
      volatile acidity
      citric acid
      residual sugar
      chlorides
      free sulfur dioxide
      total sulfur dioxide
      density
      pH
      sulphates
      alcohol
      quality

      count
      4898.000000
      4898.000000
      4898.000000
      4898.000000
      4898.000000
      4898.000000
      4898.000000
      4898.000000
      4898.000000
      4898.000000
      4898.000000
      4898.000000

      mean
      6.854788
      0.278241
      0.334192
      6.391415
      0.045772
      35.308085
      138.360657
      0.994027
      3.188267
      0.489847
      10.514267
      5.877909

      std
      0.843868
      0.100795
      0.121020
      5.072058
      0.021848
      17.007137
      42.498065
      0.002991
      0.151001
      0.114126
      1.230621
      0.885639

      min
      3.800000
      0.080000
      0.000000
      0.600000
      0.009000
      2.000000
      9.000000
      0.987110
      2.720000
      0.220000
      8.000000
      3.000000

      25%
      6.300000
      0.210000
      0.270000
      1.700000
      0.036000
      23.000000
      108.000000
      0.991723
      3.090000
      0.410000
      9.500000
      5.000000

      50%
      6.800000
      0.260000
      0.320000
      5.200000
      0.043000
      34.000000
      134.000000
      0.993740
      3.180000
      0.470000
      10.400000
      6.000000

      75%
      7.300000
      0.320000
      0.390000
      9.900000
      0.050000
      46.000000
      167.000000
      0.996100
      3.280000
      0.550000
      11.400000
      6.000000

      max
      14.200000
      1.100000
      1.660000
      65.800000
      0.346000
      289.000000
      440.000000
      1.038980
      3.820000
      1.080000
      14.200000
      9.000000

We can see that the data for both red and white wine samples take the same format.

```python
red.describe()
```

      fixed acidity
      volatile acidity
      citric acid
      residual sugar
      chlorides
      free sulfur dioxide
      total sulfur dioxide
      density
      pH
      sulphates
      alcohol
      quality

      count
      1599.000000
      1599.000000
      1599.000000
      1599.000000
      1599.000000
      1599.000000
      1599.000000
      1599.000000
      1599.000000
      1599.000000
      1599.000000
      1599.000000

      mean
      8.319637
      0.527821
      0.270976
      2.538806
      0.087467
      15.874922
      46.467792
      0.996747
      3.311113
      0.658149
      10.422983
      5.636023

      std
      1.741096
      0.179060
      0.194801
      1.409928
      0.047065
      10.460157
      32.895324
      0.001887
      0.154386
      0.169507
      1.065668
      0.807569

      min
      4.600000
      0.120000
      0.000000
      0.900000
      0.012000
      1.000000
      6.000000
      0.990070
      2.740000
      0.330000
      8.400000
      3.000000

      25%
      7.100000
      0.390000
      0.090000
      1.900000
      0.070000
      7.000000
      22.000000
      0.995600
      3.210000
      0.550000
      9.500000
      5.000000

      50%
      7.900000
      0.520000
      0.260000
      2.200000
      0.079000
      14.000000
      38.000000
      0.996750
      3.310000
      0.620000
      10.200000
      6.000000

      75%
      9.200000
      0.640000
      0.420000
      2.600000
      0.090000
      21.000000
      62.000000
      0.997835
      3.400000
      0.730000
      11.100000
      6.000000

      max
      15.900000
      1.580000
      1.000000
      15.500000
      0.611000
      72.000000
      289.000000
      1.003690
      4.010000
      2.000000
      14.900000
      8.000000

We shuffle and normalise the data such that each feature takes a value in \[0,1\], as does the quality we seek to predict. We assue that our model was trained on white wine samples, which therefore forms the reference distribution, and that red wine samples can be considered to be drawn from a drifted distribution.

```python
white, red = np.asarray(white, np.float32), np.asarray(red, np.float32)
n_white, n_red = white.shape[0], red.shape[0]

col_maxes = white.max(axis=0)
white, red = white / col_maxes, red / col_maxes
white, red = white[np.random.permutation(n_white)], red[np.random.permutation(n_red)]
X = white[:, :-1]
X_corr = red[:, :-1]
```

Although it may not be necessary on this relatively low-dimensional data for which individual features are semantically meaningful, we demonstrate how a preprocessing stage can be defined to project raw data onto a lower dimensional representation which more concisely captures the factors of variation in the data. As not to bias the detector it is necessary to learn the projection using a split of the data which isn't then passed as reference data. We additionally split off some white wine samples to act as undrifted data during deployment.

```python
X_train = X[:(n_white//2)]
X_ref = X[(n_white//2):(3*n_white//4)]
X_h0 = X[(3*n_white//4):]

X_train_ds = torch.utils.data.TensorDataset(torch.tensor(X_train), torch.tensor(X_train))
X_train_dl = torch.utils.data.DataLoader(X_train_ds, batch_size=32, shuffle=True, drop_last=True)
```

Now we define and fit an autoencder so that we can use the encoder as a preprocessing function which projects the 11-D data onto a 2-D representation.

```python
encoder = nn.Sequential(
    nn.Linear(11, 16),
    nn.ReLU(),
    nn.Linear(16, 2)
)
decoder = nn.Sequential(
    nn.Linear(2, 16),
    nn.ReLU(),
    nn.Linear(16, 11)
)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
ae = nn.Sequential(encoder, decoder).to(device)

trainer(ae, nn.MSELoss(), X_train_dl, device, torch.optim.Adam, learning_rate=0.001, epochs=10)
```

    /home/oliver/Projects/alibi-detect/.venv/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
      return torch._C._cuda_getDeviceCount() > 0
    Epoch 1/10: 100%|██████████| 76/76 [00:00<00:00, 161.73it/s, loss=0.104]
    Epoch 2/10: 100%|██████████| 76/76 [00:00<00:00, 191.40it/s, loss=0.0121]
    Epoch 3/10: 100%|██████████| 76/76 [00:00<00:00, 179.78it/s, loss=0.00488]
    Epoch 4/10: 100%|██████████| 76/76 [00:00<00:00, 172.70it/s, loss=0.00453]
    Epoch 5/10: 100%|██████████| 76/76 [00:00<00:00, 151.17it/s, loss=0.00548]
    Epoch 6/10: 100%|██████████| 76/76 [00:00<00:00, 175.39it/s, loss=0.00517]
    Epoch 7/10: 100%|██████████| 76/76 [00:00<00:00, 185.24it/s, loss=0.00426]
    Epoch 8/10: 100%|██████████| 76/76 [00:00<00:00, 136.51it/s, loss=0.00441]
    Epoch 9/10: 100%|██████████| 76/76 [00:00<00:00, 155.20it/s, loss=0.00467]
    Epoch 10/10: 100%|██████████| 76/76 [00:00<00:00, 211.75it/s, loss=0.00448]

The autoencoder was fit to data from the reference distribution (white wine samples) and should therefore more accurately reconstruct white wine samples than red.

```python
ae = ae.eval()
ae_fn = encompass_batching(ae, backend='pytorch', batch_size=32)
recon_ref = ae_fn(X_ref)
recon_corr = ae_fn(X_corr)

ref_mse = np.square(recon_ref - X_ref).mean()
corr_mse = np.square(recon_corr - X_corr).mean()

print(f'MSE when reconstructing unseen white wine samples: {ref_mse}')
print(f'MSE when reconstructing unseen red wine samples: {corr_mse}')
```

    MSE when reconstructing unseen white wine samples: 0.005474736448377371
    MSE when reconstructing unseen red wine samples: 0.024097805842757225

Hopefully the learned preprocessing step has learned a projection such that in the lower dimensional space the two samples are distinguishable.

```python
encoder_fn = encompass_batching(encoder, backend='pytorch', batch_size=32)
enc_h0 = encoder_fn(X_h0)
enc_h1 = encoder_fn(X_corr)

plt.scatter(enc_h0[:,0], enc_h0[:,1], alpha=0.2, color='green', label='white wine')
plt.scatter(enc_h1[:,0], enc_h1[:,1], alpha=0.2, color='red', label='red wine')
plt.legend(loc='upper right')
plt.show()
```

Now we can define our online drift detector. We specify an expected run-time (in the absence of drift) of 50 time-steps, and a window size of 10 time-steps. Upon initialising the detector thresholds will be computed using 2500 boostrap samples. These values of `ert`, `window_size` and `n_bootstraps` are lower than a typical use-case in order to demonstrate the average behaviour of the detector over a large number of runs in a reasonable time.

```python
from alibi_detect.cd import MMDDriftOnline

ert = 50
window_size = 10

cd = MMDDriftOnline(
    X_ref, ert, window_size, backend='pytorch', preprocess_fn=encoder_fn, n_bootstraps=2500
)
```

    No GPU detected, fall back on CPU.
      1%|▏         | 32/2500 [00:00<00:07, 312.22it/s]Generating permutations of kernel matrix..
    100%|██████████| 2500/2500 [00:07<00:00, 342.63it/s]
    Computing thresholds: 100%|██████████| 10/10 [00:04<00:00,  2.18it/s]

We now define a function which will simulate a single run and return the run-time. Note how the detector acts on single instances at a time, the run-time is considered as the time elapsed after the test-window has been filled, and that the detector is stateful and must be reset between detections.

```python
def time_run(cd, X, window_size):
    n = X.shape[0]
    perm = np.random.permutation(n)
    t = 0
    cd.reset()
    while True:
        pred = cd.predict(X[perm[t%n]])
        if pred['data']['is_drift'] == 1:
            return t
        else:
            t += 1
```

Now we look at the distribution of run-times when operating on the held-out data from the reference distribution of white wine samples. We report the average run-time, however note that the targeted run-time distribution, a Geometric distribution with mean `ert`, is very high variance so the empirical average may not be that close to `ert` over a relatively small number of runs. We can see that the detector accurately targets the desired Geometric distribution however by inspecting the linearity of a [Q-Q plot](https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot).

```python
n_runs = 250
times_h0 = [time_run(cd, X_h0, window_size) for _ in range(n_runs)]
print(f"Average run-time under no-drift: {np.mean(times_h0)}")
_ = scipy.stats.probplot(np.array(times_h0), dist=scipy.stats.geom, sparams=1/ert, plot=plt)
```

If we run the detector in an identical manner but on data from the drifted distribution of red wine samples the average run-time is much lower.

```python
n_runs = 250
times_h1 = [time_run(cd, X_corr, window_size) for _ in range(n_runs)]
print(f"Average run-time under drift: {np.mean(times_h1)}")
```

    Average run-time under drift: 5.456

## Online detection with LSDD and TensorFlow

Here we address the same problem but using the least squares density difference (LSDD) as the two-sample distance in a manner similar to [Bu et al. (2017)](https://ieeexplore.ieee.org/abstract/document/7890493). The LSDD between two distributions $p$ and $q$ on $\mathcal{X}$ is defined as $$LSDD(p,q) = \int_{\mathcal{X}} (p(x)-q(x))^2 \,dx$$ and also has an empirical estimate $\widehat{LSDD}(\{X_i\}_{i=1}^N, \{Y_i\}_{i=t}^{t+W})$ that can be updated at low cost as the test window is updated to $\{Y_i\}_{i=t+1}^{t+1+W}$.

We additionally show that TensorFlow can also be used as the backend and that sometimes it is not necessary to perform preprocessing, making definition of the drift detector simpler. Moreover, in the absence of a learned preprocessing stage we may use all of the reference data available.

```python
X_ref = np.concatenate([X_train, X_ref], axis=0)
```

And now we define the LSDD-based online drift detector, again with an `ert` of 50 and `window_size` of 10.

```python
from alibi_detect.cd import LSDDDriftOnline

cd = LSDDDriftOnline(
    X_ref, ert, window_size, backend='tensorflow', n_bootstraps=2500,
)
```

    Computing thresholds: 100%|██████████| 9/9 [00:10<00:00,  1.12s/it]

We run this new detector on the held out reference data and again see that in the absence of drift the distribution of run-times follows a Geometric distribution with mean `ert`.

```python
n_runs = 250
times_h0 = [time_run(cd, X_h0, window_size) for _ in range(n_runs)]
print(f"Average run-time under no-drift: {np.mean(times_h0)}")
_ = scipy.stats.probplot(np.array(times_h0), dist=scipy.stats.geom, sparams=1/ert, plot=plt)
```

And when drift has occured the detector is very fast to respond.

```python
n_runs = 250
times_h1 = [time_run(cd, X_corr, window_size) for _ in range(n_runs)]
print(f"Average run-time under drift: {np.mean(times_h1)}")
```

    Average run-time under drift: 4.176


# Model Project

This project will create and simulate an Small Open Economy using an AS-AD-model. We examine the effects on aggregate demand and supply from changes in interest rates both domestic and abroad, and furhter examine effects from changes in the effective exchange rate.

The Model is structured the following way

 1. Defining the AS-AD Model
 2. Defining the Aggregate Demand (AD)
 3. Defining Money Supply
 4. Defining the Long Run Aggregate Supply (LRAS)
 5. Defining the Short Run Aggregate Supply (LRAS)
 6. Combining all elements
 7. Simulating a shock to domestic and foreing interest rate.

# Defining the Aggregate Demand

//DEFINER AD teoretisk, aka ligninger og så videre

```python
#Importing Packeges used
import numpy as np
import matplotlib.pyplot as plt
from scipy import optimize
import sympy as sm
```

```python
#Creating the AD

class AD
    #Using an ___init___ method, we define the parameters of the model
    def ___init___ ( )

    #We assign values to the parameters
    self.a =

    #HER SKAL INDGÅ 2 FUNCTIONER, EN FOR MONEY SUPPLY OG EN FOR MONEY MULTIPLIER

    def AD_curve(self)
        #Kald på self.parametrene

        #Return AD-kurven

```


# Example Model Project: the Train Illusion

**By Neuromatch Academy**

__Content creators:__ Marius t'Hart, Megan Peters, Paul Schrater, Gunnar Blohm

__Production editors:__ Spiros Chavlis

**Disclaimer**: this is a "toy" model used to demonstrate the [10 step procedure of how-to-model](https://doi.org/10.1523/ENEURO.0352-19.2019). It is not meant to be state of the art research.

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Phenomenon
*Part of Steps 1-2*

The train illusion occurs when sitting on a train and viewing another train outside the window. Suddenly, the other train *seems* to move, i.e. you experience visual motion of the other train relative to your train. But which train is actually moving?

Often people have the wrong percept. In particular, they think their own train might be moving when it's the other train that moves; or vice versa. The illusion is usually resolved once you gain vision of the surroundings that lets you disambiguate the relative motion; or if you experience strong vibrations indicating that it is indeed your own train that is in motion.

---
# Question

*Part of Step 1*

We asked the following (arbitrary) question for our demo project: "How do noisy vestibular estimates of motion lead to illusory percepts of self motion?"

---
# Background
*Part of Step 2*

You have learned all about the vestibular system in the Intro video. This is also where you would do a literature search to learn more about what's known about self-motion perception and vestibular signals. You would also want to examine any attempts to model self-motion, perceptual decision making and vestibular processing.

---
# Ingredients

*Part of step 3*

We determined that we probably needed the following ingredients for our model:
* Vestibular input: $v(t)$
* Binary decision output: $d$ - time dependent?
* Decision threshold: $\theta$
* A filter (maybe running average?): $f$
* An integration mechanism to get from vestibular acceleration to sensed velocity: $\int$

---
# Hypotheses

*Part of step 4*

Our main hypothesis is that the strength of the illusion has a linear relationship to the amplitude of vestibular noise.

Mathematically, this would write as $S = k \cdot N$, where $S$ is the illusion strength and $N$ is the noise level, and $k$ is a free parameter.

>we could simply use the frequency of occurance across repetitions as the "strength of the illusion"

We would get the noise as the standard deviation of $v(t)$, i.e., $N = \mathbb{E}[v(t)^2]$, where $\mathbb{E}[\cdot]$ stands for the expected value.

Do we need to take the average across time points?

> doesn't really matter because we have the generative process, so we can just use the $\sigma$ that we define

---
# Selected toolkit

*Part of step 5*

We chose to use a [Drift-Diffusion Model (DDM)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2474742/) because it is a well-established framework that allows us to model decision making in the case of 2 alternative choices (here: self-motion vs. other train motion).

For our purposes simplest equation looks something like this:

\begin{align}
\dot e = \frac{de}{dt}= -c \cdot e + v \, ,
\end{align}

where $e$ is the accumulated evidence and $v$ is our vestibular input already containing the noise (so we don't need to add more noise?). $c$ is the leakage constant, i.e., $c=0$ means perfect integration; $c=1$ means no integration (perfect leakage).

---
# Model draft

*Part of step 6*

Brainstorming on the whiteboard, we came up with this...

---
# Model implementation

*Part of step 7*

We need at least 3 functions:
1. vestibular signal generator
2. integrator (or drift-diffusion mechanism)
3. decision mechanism (threshold comparison)

**Note:** we did not add a filter (yet). We're not sure if we need one...

So let's go step by step... first we need to get set up...

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
!pip install pandas --quiet
plt.style.use('dark_background')
```

## 1. Vestibular signal generator

```python
def vestibular_signal(sig, mov):
  """
  Computes a vestibular signal that corresponds to the acceleration of the
  train with different amplitudes of noise
  Args:
    sig: scalar SD of noise
    mov: 0 means no self-motion; 1 means self-motion (or scaling or motion signal)
  Returns: vector array of vestibular signal v
  """
  # create white noise series for 10s with 1ms resolution
  x = np.linspace(-7, 14, 1001)
  z = 1/(1 + np.exp(-x))
  noise = norm.rvs(size=1000)
  v = sig*noise + mov*np.diff(z)/0.001

  return v
```

Let's see if that works... (*unit test*)

```python
v = vestibular_signal(1,1)

# plot signal
t = np.linspace(0, 10, 1000)
plt.plot(t,v)
plt.xlabel("time (s)")
plt.ylabel("vestibular signal (a.u.)")
```

## 2. Integrator (DDM mechanism)

```python
def ddm(v, c):
  """
  Leaky integration of vestibular signal
  Args:
    v: array of vestibular signal
    c: leakage constant
  Outputs: time series of integrated vestibular signal (with leakage)
    = evidence e
  """
  e = np.random.normal(0, 0.1)
  E = []
  for i in range(len(v)):
    e += -c*e + v[i]*0.001
    E.append(e)
  return E
```

Let's test the DDM function... (*unit test*)

```python
e = ddm(v, c=0.001)

# plot result
plt.plot(t, e)
plt.xlabel("time (s)")
plt.ylabel("integrated evidence")
plt.show()
```

## 3. Thresholding (decision mechanism)

```python
def threshold(e, thr):
  """
  Thresholding of motion evidence
  Args:
    motion evidence: e (array)
    threshold: thr
  Output: decision d if threshold was reached
  """
  d = any(np.array(e) > thr)*1
  return d
```

Now let's test this function... (*unit test*)

```python
d = threshold(e, .6)
print(d)
```

    1

## Assembling the model

```python
def run_model(sig, c, thr, mov):
  """
  runs the full model to simulate self-motion decision, e.g., for train illusion
  Args:
    sig: SD of vestibular noise
    c: leakage constant
    thr: decision threshold
    mov: self-motion? ) no selfmotion; 1 self-motion
  Output: decision d (0: no self-motion; 1: self-motion)
  """
  v = vestibular_signal(sig, mov)
  e = ddm(v, c)
  d = threshold(e, thr)
  return d
```

Let's run the model and see if it works...

```python
d = run_model(200, 0.001, 0.8, 1)
print(d)
```

    1

---
# Model completion

*Part of step 8*

So the model seems to work. Running different parameters gives us different results. Are we done?
* **can we answer our question**: yes, in our model the illusion arises because integrating very noisy vestibular signals representing motion evidence sometimes accumulate to a decision threshold and sometimes do not reach that threshold.
* **can we speak to our hypothesis**: yes, we can now simulate different trials with different noise levels (and leakage and thrshold parameters) and evaluate the hypothesized linear relationship between  vestibular noise and how often our perceptual system is fooled...
* **does the model reach our goals**: yes, we wanted to generate a mechanistic model to be able to make some specific predictions that can then be tested experimentally later...

---
# Model evaluation & testing

*Part of step 9*

Ok, so we still need to actually evaluate and test our model performance. Since this is a conceptual model and we don't have actual data (yet), we will evaluate how our model behaves as a function of the 3 parameters. If we had data with different conditions, we could try to fit the model to the data and evaluate the goodness of fit, etc. If other alterative models existed, we could evaluate our model against those alternatives too.

So let's run out model in different parameter regimes and analyze the result to get some insight into the model performance

```python
import itertools  # to automatically generate possible combinations of parameters

# define parameter list
params = {
    'sig': np.linspace(1, 21, 5)**2,
    'c': np.exp(np.linspace(-10, -1, 5)),
    'thr': np.linspace(0, 2, 5),
    'mov': np.linspace(0, 1, 2),
    }

# run combination of parameters
keys = list(params)
D = []
for i in range(0,100):
  for values in itertools.product(*map(params.get, keys)):
    d = run_model(**dict(zip(keys, values)))
    temp = list(values)
    temp.append(d)
    D.append(temp)
```

Now let's explicitly test our hypothsis for different parameter combinations...

```python
# want data frames:
import pandas as pd
df = pd.DataFrame(D, columns=['Sig', 'c', 'Thr', 'Mov', 'Decisions'])
# multi panel layout:
axs = plt.figure(figsize=(12,12), constrained_layout=True).subplots(5, 5)
# plot for movement absent/present
Mov_s = np.unique(df['Mov'])
# plot for leakage parameter & threshold values:
c_s = np.unique(df['c'])
Thr_s = np.unique(df['Thr'])
# plot for data for both movement condition for each leakage/threshold combination
Sig_s = np.unique(df['Sig'])
for Thr_n in range(len(Thr_s)):
  for c_n in range(len(c_s)):
    subdf0 = df[(df.Mov == 0) & (df.c == c_s[c_n]) & (df.Thr == Thr_s[Thr_n])].groupby(['Sig'])['Decisions'].mean()
    subdf1 = df[(df.Mov == 1) & (df.c == c_s[c_n]) & (df.Thr == Thr_s[Thr_n])].groupby(['Sig'])['Decisions'].mean()
    im0 = axs[Thr_n, c_n].plot(Sig_s, subdf0, label="no motion")
    im1 = axs[Thr_n, c_n].plot(Sig_s, subdf1, label="motion")
    axs[Thr_n, c_n].set_title(f"Thr = {Thr_s[Thr_n]}; c = {c_s[c_n]:.4f}")
    axs[Thr_n, c_n].set_ylim(0, 1.1)
    axs[Thr_n, c_n].set_xlim(0, 450)
axs[4, 2].set_xlabel("Noise level $\sigma$")
axs[2, 0].set_ylabel("Proportion motion judgment")
axs[3, 1].set_facecolor('grey')
axs[4, 4].legend()
plt.show()
```

There seems to be some parameter redundancy, i.e., we could chose different parameter combinations to make the model do something sensible...

But it looks like $c=0.0004$ works well for $\theta = 1.5$ and $\sigma=50$ (highlighted plot). Lets run a few trials on that to analyze those results more specifically...

```python
# run "best" parameter combination
sig = 50
c = 0.0004
thr = 1.5
d0 = []
d1 = []
for i in range(0, 1000):
  d0.append(run_model(sig, c, thr, 0))
  d1.append(run_model(sig, c, thr, 1))
print(f"\n Motion detected for no-motion: {sum(d0)/10}% and motion: {sum(d1)/10}%")
```

     Motion detected for no-motion: 27.4% and motion: 59.2%

This does indeed result in roughly 50% likelihood of experiencing the illusion both ways.

Finally, let's explicitly evaluate our hypothesis...

```python
sig = np.linspace(1, 201, 20)
c = 0.0004
thr = 1.5
D0 = []
D1 = []
for s in range(len(sig)):
  d0 = []
  d1 = []
  for i in range(0, 100):
    d0.append(run_model(sig[s], c, thr, 0))
    d1.append(run_model(sig[s], c, thr, 1))
  D0.append(sum(d0) / 100)
  D1.append(sum(d1) / 100)
```

```python
plt.plot(sig, D0, label="no motion")
plt.plot(sig, D1, label="motion")
plt.xlabel("Noise level")
plt.ylabel("% motion decisions")
plt.legend()
plt.show()
```

Our **hypothesis** of linear increase of illusion strength with noise only holds true in a limited range of noise... It's monotonic but saturating of course...

**And regarding our original question**: it is really the noise that pushes the integrated signal over the threshold. The less leaky the integration and the lower the threshold, the more motion decisions we get...

---
# Summary
*Part of Step 10*

Let's write a simple abstract following the guidelines...

**A. What is the phenomena**?  Here summarize the part of the phenomena which your modeling addresses.

_The "train illusion" occurs when sitting in a stationary train and experiencing relative visual motion of an adjacent train outside the window; sometimes we feel like we're moving even if we're not (and vice versa). Previous literature has suggested that vestibular signals are used to disambiguate self-motion from motion of an adjacent object._

**B. What is the key scientific question?**:  Clearly articulate the question which your modeling tries to answer.

_How noisy vestibular estimates of motion lead to illusory percepts of self motion is currently unknown._

**C. What was our hypothesis?**:  Explain the key relationships which we relied on to simulate the phenomena.

_We hypothesized that noisy vestibular signals are integrated leading the brain to decide whether self-motion is occurring or not, and that larger noise is linearly associated with more frequent errors in self-motion judgment._

**D. How did your modeling work?** Give an overview of the model, it's main components, and how the modeling works.  ''Here we ... ''

_To investigate this hypothesis, we constructed a drift diffusion model and simulated self-motion decisions under varying noise conditions, when true self motion was occurring or not._

**E. What did you find? Did the modeling work?** Explain the key outcomes of your modeling evaluation.

_We observed that higher noise did indeed lead to more frequent errors in self-motion perception but this relationship was not linear._

**F. What can you conclude?** Conclude as much as you can _with reference to the hypothesis_, within the limits of the modeling.

_We conclude that accumulated noisy vestibular information can explain the occurrence of the train illusion, and the higher the noise (or the lower the signal-to-noise ratio), the more frequently such illusions will occur._

**G. What are the limitations and future directions?** What is left to be learned? Briefly argue the plausibility of the approach and/or what you think is essential that may have been left out.

_Future research should investigate whether trial-by-trial variations of noisy vestibular signals actually correlate with self-motion judgments._

>If we put this all in one paragraph, we have our final complete abstract. But, first, do not include the letters in _your_ abstract, and second, we did paraphrase the answers a little so they fit together.

## Abstract
(A) The "train illusion" occurs when sitting in a stationary train and experiencing relative visual motion of an adjacent train outside the window; sometimes we feel like we're moving even if we're not (and vice versa). Previous literature has suggested that vestibular signals are used to disambiguate self-motion from motion of an adjacent object. (B) How noisy vestibular estimates of motion lead to illusory percepts of self motion is currently unknown. (C) We hypothesized that noisy vestibular signals are integrated leading the brain to decide whether self-motion is occurring or not, and that larger noise is linearly associated with more frequent errors in self-motion judgment. (D) To investigate this hypothesis, we constructed a drift diffusion model and simulated self-motion decisions under varying noise conditions, when true self motion was occurring or not. (E) We observed that higher noise did indeed lead to more frequent errors in self-motion perception but this relationship was not linear. (F) We conclude that accumulated noisy vestibular information can explain the occurrence of the train illusion, and the higher the noise (or the lower the signal-to-noise ratio), the more frequently such illusions will occur. (G) Future research should investigate whether trial-by-trial variations of noisy vestibular signals actually correlate with self-motion judgments.

---
# Final thoughts

Note that the model we built here was extremely simple and used artificial data on purpose. It allowed us to go through all the steps of building a model, and hopefully you noticed that it is not always a linear process, you will go back to different steps if you hit a roadblock somewhere.

There are many issues that we did not address in this model. However, if you're interested in how to actually approach modeling a similar phenomenon in a probabilistic way, we encourage you to read the paper by [Dokka et al., 2019](https://doi.org/10.1073/pnas.1820373116), where the authors model how judgments of heading direction are influenced by objects that are also moving.


# NOTE
This notebook doesn't work yet. I have previously written my own version of `lambdify` [here](https://github.com/bjodah/sym/blob/222d1c6a3d1db03c67e8467ff562d3ac0f124616/sym/_sympy_Lambdify.py#L214).
Don't know if that's the path to go, or wait for [next release of numba](https://github.com/numba/numba/issues/2328#issuecomment-303382911).

In this notebook we will use `numba` the increase the performance of our callbacks produced by lambdify in SymPy.

```python
import json
import numpy as np
import sympy as sym
from scipy2017codegen.odesys import ODEsys
from scipy2017codegen.chem import mk_rsys
```

The `ODEsys` class and convenience functions from previous notebook (35) has been put in two modules for easy importing. Recapping what we did last:

```python
watrad_data = json.load(open('../scipy2017codegen/data/radiolysis_300_Gy_s.json'))
watrad = mk_rsys(ODEsys, **watrad_data)
tout = np.logspace(-6, 3, 200)  # close to one hour of operation
c0 = {'H2O': 55.4e3, 'H+': 1e-4, 'OH-': 1e-4}
y0 = [c0.get(symb.name, 0) for symb in watrad.y]
```

```python
%timeit yout, info = watrad.integrate_odeint(tout, y0)
```

so that is the benchmark to beat.

```python
from numba import njit
watrad_numba = mk_rsys(ODEsys, **watrad_data, lambdify=lambda *args: njit(sym.lambdify(*args, modules="numpy")))
watrad_numba.integrate_odeint(tout, y0)
```

```python
%timeit watrad_numba.integrate_odeint(tout, y0)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

Just to see that everything looks alright:

```python
fig, ax = plt.subplots(1, 1, figsize=(14, 6))
watrad_numba.plot_result(tout, *watrad_numba.integrate_odeint(tout, y0), ax=ax)
ax.set_xscale('log')
ax.set_yscale('log')
```

```python

```


Copyright: Thomas Klinder

# Increase Your Property Value and Stretch Your House Buying Budget Further

**Phase 2 Final Project**

* Student name: Elena Kazakova
* Student pace: full time
* Cohort: DS02222021
* Scheduled project review date/time: 05/07/2021
* Instructor name: James Irving
* Blog post URL: The Power of Color

## Table of Contents

*Click to jump to matching Markdown Header.*

- **[Introduction](#INTRODUCTION)
**
- **[Obtain](#Obtain)**

- **[Scrub](#Scrub)**

- **[Explore](#Explore)**

- **[Model](#Model)**

- **[iNterpret](#iNterpret)**

- **[Conclusions/Recommendations](#Conclusions-and-Recommendation)
**
___

# Introduction

## Business Problem
This project is the Inference Analysis project of King County, WA house prices, and various factors that might affect the sales price.

This study aims to build a model(s) of house sale prices depending on the features of the property in the dataset provided. This information can be helpful for house owners, house buyers, and real estate agents in the county.

# Obtain

## Data Understanding

The dataset used in this project has been downloaded from [KAGGLE site]( https://www.kaggle.com/harlfoxem/housesalesprediction?select=kc_house_data.csv). The dataset includes the information about properties sold in King County of Washington State between May 2014 and May 2015. The area consists of Seattle city area but does not include the inner city. The dataset consists of 21 dependent and independent variables and 21597 records.

### Importing Python tools and utilities

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import folium
import json

import plotly.express as px

import statsmodels.api as sm
import statsmodels.formula.api as smf
import statsmodels.stats.api as sms
import sklearn.metrics as metrics
#import plotly.graph_objects as go

import scipy.stats as stats

import math
#import pickle
import scipy.stats

from matplotlib import style

from statsmodels.formula.api import ols
from statsmodels.stats.outliers_influence import variance_inflation_factor

#from pandasql import sqldf

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
#from sklearn.preprocessing import OneHotEncoder, PolynomialFeatures, StandardScaler, RobustScaler, MinMaxScaler
from sklearn.preprocessing import OneHotEncoder, PolynomialFeatures, RobustScaler, MinMaxScaler

from sklearn.feature_selection import RFE
from sklearn.metrics import make_scorer
from sklearn.model_selection import cross_val_score
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

from scipy.stats import pearsonr
from IPython.display import display, Math, Latex
from folium import plugins
from plotly.subplots import make_subplots

#from mlxtend.evaluate import bias_variance_decomp

from warnings import filterwarnings
filterwarnings('ignore')

%matplotlib inline
```

### Functions used

```python
def count_unique_records(df):
    unique_records=[]
    for column in df.columns:
        n = df[column].nunique()
        unique_records.append((column,n))
    print(unique_records)
    return None

def count_dups_field(field1):
    dups = df.pivot_table(index = [field1], aggfunc ='size')
    return dups

def count_dups_fields(field1, field2):
    dups = df.pivot_table(index = [field1, field2], aggfunc ='size')
    return dups

def remove_columns(df, y_columns=['price'], x_columns=[], exclude_columns=[], add_constant=True):

    if x_columns==[]:
        x_columns=list(df.drop(columns=y_columns, axis=1))

    [x_columns.remove(columns) for columns in exclude_columns]

    df_x=df[x_columns]
    df_y=df[y_columns]

    return df_x, df_y

# Function to convert geo coordinates to distance from center. I am using coordinates
def distance_from_center(lat_coord,lon_coord):
    R = 3959.999

# I am using geo coordinates of Seattle 47.6062° N, 122.3321° W, from Wikipedia

    lat1 = math.radians(47.6062)
    lon1 = math.radians(122.3321)
    lat2 = math.radians(lat_coord)
    lon2 = math.radians(lon_coord)

    dlon = lon2 - lon1
    dlat = lat2 - lat1

    a = math.sin(dlat / 2)**2 + math.cos(lat1) * math.cos(lat2) * math.sin(dlon / 2)**2
    c = 2 * math.atan2(math.sqrt(a), math.sqrt(1 - a))
    d=R*c
    return round(d,1)

def jointplot(df):
    sns.set_style('white')
    for col in df.columns:
        g=sns.jointplot(x=col, y='price', data=df, size=5, kind='reg', marginal_ticks=True,
                        joint_kws={'line_kws':{'color':'green'}}, height=15, space=0.7)
        name=col
        R2,p= scipy.stats.pearsonr(x=df[col], y=df.price)
        g.fig.suptitle('For {}: R2 coefficient is {}, p-value is {}'.format(name, round(R2,4),p))
        g.fig.tight_layout()
        g.fig.subplots_adjust(top=0.85)

    return None

def r2_p(df):
    for col in df.columns:
        name=col
        R2,p= scipy.stats.pearsonr(x=df[col], y=df.price)
        print('For {}: R2 coefficient is {}, p-value is {}'.format(name, round(R2,4),p))
    return None

# This is a snippet from https://www.analyticsvidhya.com/blog/2020/03/what-is-multicollinearity/
def calc_vif(X):

    # Calculating VIF
    vif = pd.DataFrame()
    vif["variables"] = X.columns
    vif["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

    return(vif)

# This is a snippet from https://atmamani.github.io/cheatsheets/seaborn/seaborn_cheat_sheet_1/
def distribution(column):
    col_mean = column.mean()
    col_sd = column.std()
    skew_val = stats.skew(column, bias=False)
    kurt_val = stats.kurtosis(column,bias=False)

    ax = sns.distplot(column, kde_kws={"color": "r", "lw": 2, "label": "KDE", "bw_adjust": 3})

    ax.axvline(x=col_mean, color='black', linestyle='dashed')

    ax.axvline(x=col_mean + col_sd, color='red', linestyle='dotted')
    ax.axvline(x=col_mean - col_sd, color='red', linestyle='dotted')

    ax.set_title('$\mu = {}$ | $\sigma = {}$ | Skew = {} | Kurtosis = {}'.
                 format(round(col_mean, 2), round(col_sd, 2), round(skew_val,2), round(kurt_val,2)))

    plt.subplots_adjust(top=0.5)
    plt.tight_layout()

    return None

def boxen_plot(df,colname):
    ax = sns.catplot(x=df[colname], y=df.price/1000000, kind="boxen",
                 data=df.sort_values(colname), height=7, aspect=8/5)
    ax.set_xticklabels(fontsize=12)
    ax.set_yticklabels(fontsize=12)
    plt.ylabel('Price in millions', fontsize=15)
    plt.xlabel(colname,fontsize=15)
    plt.grid()
    plt.show()
    return None

#Zipcode choropleth maps with average values per a zipcode (King County)
def map_choropleth_zip(df, column, title, column_name):
    fig=px.choropleth_mapbox(data_frame=df, locations='zipcode', geojson=KC_zip_json, color=column,
                         mapbox_style='open-street-map', zoom=8.5, height=900, featureidkey='properties.ZCTA5CE10',
                        center={'lat': 47.403768, 'lon': -122.005863}, opacity=0.4,
                        color_continuous_scale=px.colors.sequential.YlOrRd,
                        title=title,
                        template = "plotly_dark",
                        labels={
                            column: column_name})
    fig.update_layout(
    font_family="Arial",
    font_size=16,
    font_color="white",
    title_font_family="Arial",
    title_font_color="white",
    title_font_size=20)

    fig.update_layout(
    title={
        'y':0.98,
        'x':0.5,
        'xanchor': 'center',
        'yanchor': 'top',
    })

    fig.show()
    return None

```

### Importing data

```python
# Importing raw data
df=pd.read_csv('data/kc_house_data.csv')
pd.set_option('display.width', 1000)
df.head()
```

```python
df.info()
```

```python
# Displaying tuples of fields and unique records in them
count_unique_records(df)
df.nunique()
```

```python
# Finding all duplicate IDs

df_a=pd.DataFrame(count_dups_field('id'))
df_a.columns=['count']
df_a = df_a[df_a['count'] > 1].sort_values('count', ascending=False)
df_a.reset_index(level=0, inplace=True)
df_a
```

```python
# Listing duplicate IDs and associated dataset records to find out the reason they are in the file twice

df_dup=pd.DataFrame()
for i in range(len(df_a)):
    df_dup=df_dup.append(df[df.id==df_a.id[i]], ignore_index=True)
df_dup
```

*The inspection of the records shows that the records with duplicate IDs have different sale dates and sale prices. However, all other features remain the same. Records with the same set of predictors but different prices would introduce additional "noise" to the data. Because there are only 353 records with that problem, I decided to drop them from the dataset.*

```python
#listing datatype, number of null values, min and max values in the fields of the dataset

fields1=['bedrooms','bathrooms','floors','waterfront', 'view','condition','grade']
fields2=['sqft_lot15', 'sqft_living15','yr_renovated','yr_built','sqft_above','sqft_living','sqft_lot']
for column in df.columns:
    type_=df[column].dtypes
    num_nulls=df[column].isna().sum()
    min_=0
    max_=0
    unique_=[0]
    if column in fields1:
        unique_=df[column].unique()
        unique_.sort()
    else:
        if column in fields2:
            min_=df[column].min()
            max_=df[column].max()
        else:
            continue
    print('Column name:', column)
    print('Type:', type_)
    print('Number of null values', num_nulls)
    print('Unique values:',unique_)
    print('Min value: ', min_, 'Max value:', max_)
    print('***********************************')
```

### Description of the fields

The file has 21597 records with 21 columns, out of which 11 columns have integer values, 8 are real numbers, and 2 are strings.

    The annotation to the fields and associated data
 (link to the definitions [here](https://github.com/emilyageller/king_county_house_prices))
* id: Unique ID for each home sold

    1. no NULL values

    2.integer numbers

    3.176 duplicate records
>353 rows to be dropped

* **date:** Date of the sale
>no NULL values

string

>Convert to DateTime type

* **price** - Price of each home sold
>no NULL values

Real numbers

Minimum price: 78000

Maximum price: 7700000

* **bedrooms** - Number of bedrooms
>no NULL values

>Integer numbers, between 1 and 33

* **bathrooms** - Number of bathrooms, where .5 accounts for a room with a toilet but no shower
>no NULL values

>Real numbers, between 0.5 and 8.0

* **sqft_living** - Square footage of the house interior living space
>No NULL values

Integer numbers

Minimum value: 370

Maximum value: 13540

* **sqft_lot** - Square footage of the land lot
>No NULL values

Integer numbers

Minimum value: 520

Maximum value: 1651359

* **floors** - Number of floors
>no NULL values

>Real numbers, between 1.0 and 3.5

* **waterfront** - A categorical variable for whether the house was overlooking the waterfront or not
>**2376** NULL values

>Real numbers, only two values 1.0 and 0.0

>Convert to a categorical variable

Waterfront, not Waterfront

>Replace NULL values with "Missing" category

* **view** - A categorical variable describing how good the view of the property was
>**63** NULL values

>Real numbers: 1.0, 2.0, 3.0, 4.0

>Convert to a categorical variable

Poor, Fair, Good Excellent

>Replace NULL values with "Missing" category

* **condition** - A categorical variable describing the condition of the house
>no NULL values

>Integer numbers, between 1 and 5

>Convert to a categorical variable

Poor, Fair, Good, Very Good, Excellent

* **grade** - A categorical variable describing the quality of construction, from 1 to 13; 1-3 falls short of building construction and design, 7 has an average level of construction and design, and 11-13 have a high quality level of construction and design.
>no NULL values

>Integer numbers, between 3 and 13

* **sqft_above** - The square footage of the interior housing space that is above ground level
>No NULL values

Integer numbers

Minimum value: 370

Maximum value: 9410

* **sqft_basement** - The square footage of the interior housing space that is below ground level
>No NULL values

String

>Convert to integer

* **yr_built** - The year the house was initially built
>No NULL values

Integer numbers

Minimum value: 1900

Maximum value: 2015

* **yr_renovated** - The year of the last house renovation
>**3842** NULL values

>Real numbers, between 0.0 and 2015.0

>Convert to integer

* **zipcode** - What zipcode area the house is in
>no NULL values

Integer numbers, 70 unique values

>Convert to categorical variable or drop

* **lat** - Lattitude
>no NULL values

>Real numbers

* **long** - Longitude
>no NULL values

>Real numbers

* **sqft_living15** - The square footage of interior housing living space for the nearest 15 neighbors
>No NULL values

Integer numbers

Minimum value: 399

Maximum value: 6210

* **sqft_lot15** - The square footage of the land lots of the nearest 15 neighbors
>No NULL values

Integer numbers

Minimum value: 651

Maximum value: 871200

### Initial cleaning of the data

```python
df.info()
```

#### 1. Dropping duplicate rows for houses sold twice in the timeframe of the dataset

```python
#Dropping duplicate rows for houses sold twice in the timeframe of the dataset
df = df.drop_duplicates(subset='id', keep="first")
df
```

#### 2. Converting the 'date' field to DateTime formate and making sure it worked

```python
# date string to datetime type
df['date'] = pd.to_datetime(df['date'])
```

```python
#Checking if the conversion went OK
mask = (df['date'] > '9/24/2014') & (df['date'] <= '4/22/2015')
df.loc[mask]
```

#### 3. Locations of the houses with missing 'waterfront' values

```python
# Possible strategies:
# 1. Check if there are waterfront properties among neighbors within a certain distance range
# 2. Make a map and place properties with missing values on it visually
# 3. What is the longitude of the bay shore? Any house with a missing value too far away from it
#    should have their waterfront value set to 0. Hopefully, it will eliminate most of the missing values in this field
```

```python
# Leaving just coordinates of the waterfront NaN properties
mask = (df['waterfront'].isna())
df_wf_na_coord=df.loc[mask]
df_wf_na_coord.drop(df_wf_na_coord.columns[[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20]], axis=1, inplace=True)

df_wf_na_coord
```

##### 3.1 Visual assesment of the houses with Null value in the 'waterfront column

```python
# Initializing the map based on the Wikipedia location of Seattle
KCMap = folium.Map(location=[47.6171,-122.3249],  tiles='Stamen Terrain', zoom_start=9)

# for each row in the KC_house dataset with missing waterfront value,
# plot the corresponding latitude and longitude on the map
for index, row in df_wf_na_coord.iterrows():
    folium.CircleMarker((row['lat'], row['long']), radius=1, weight=2,
                        color='red', fill_color='red', fill_opacity=.5).add_to(KCMap)

display(KCMap)
# adding the heatmap
KCMap.add_child(plugins.HeatMap(data=df_wf_na_coord[['lat', 'long']], radius=20, blur=10))

```

```python
# Displaying
# the percentage of waterfront properties in the dataset with missing waterfront values,
# the percentage of waterfront properties in the full dataset
# The systemic error introduced to the full dataset if missing waterfront values would be replaced with 0

a=round((20/len(df_wf_na_coord))*100,3)

b=round((len(df[df['waterfront'] == 1])/len(df)*100),2)

c=round(round(((len(df[df['waterfront'] == 1])+20)/len(df)*100),3)-round((len(df[df['waterfront'] == 1])/len(df)*100),3),3)

print('{}%, {}%, {}%'.format(a, b, c))
```

*It is self-evident from the visuals above that the vast majority of the houses are located inland. Simple zooming in the maps allows a rough counting of alleged waterfront properties. The estimate is approximately 20 waterfront houses. It is 0.85% of all properties with no value in 'waterfront column (2353). In the primary dataset, the percentage of waterfront properties out of the total number of properties is 0.68%. The numbers above indicate that replacing the NaN values with 0 would introduce a systemic error of 0.01% to the whole system.*

Conclusion: The NULL values in the 'waterfront' column will be replaced with 0.

##### 3.2 Replacing NULL values in 'waterfront' column and converting the column to the integer datatype to make it categorical

```python
#Replacing NaN values in 'waterfront' column

df.loc[df.waterfront.isna(),'waterfront']=0
df.waterfront=df.waterfront.astype('int64')

subset_df = df[df['waterfront'] == 1]
count = len(subset_df)
print(count)
#df.info()
```

```python
df.info()
```

##### 3.3 Replacing NULL values in waterfront  and view field  using IterativeImputer

```python
# Dropping columns with str datatypes to use IterativeImputer on the rest

df_to_II=df.drop(['date','id','zipcode','sqft_basement'],axis=1)
df_to_II.info()
```

```python
# Using IterativeImputer to fill in Nan cells

imp = IterativeImputer(max_iter=10,random_state=0)
imp.fit(df_to_II)
imputed_df_to_II = imp.transform(df_to_II)
imputed_df = pd.DataFrame(imputed_df_to_II, columns=df_to_II.columns)
imputed_df.info()
```

```python
df.head(20)
```

```python
imputed_df.head(20)
```

```python
# Creating a subset of records with imputed NaN values

subset_df = imputed_df[(imputed_df['yr_renovated'] < 1934) & (imputed_df['yr_renovated'] != 0)]
count = len(subset_df)
print(count)
```

```python
# Listing Imputed NaN values in yr_renovated column
list_unique=list(df.yr_renovated.unique())

inverse_boolean_series = ~imputed_df.yr_renovated.isin(list_unique)
inverse_filtered_df = imputed_df[inverse_boolean_series]
inverse_filtered_df.yr_renovated.sort_values()
```

```python
# Listing Imputed NaN values in waterfront column
inverse_boolean_series = ~imputed_df.waterfront.isin([0,1])
inverse_filtered_df = imputed_df[inverse_boolean_series]
inverse_filtered_df.waterfront.sort_values()

```

```python
# Listing Imputed NaN values in view column
inverse_boolean_series = ~imputed_df.view.isin([0,1,2,3,4])
inverse_filtered_df = imputed_df[inverse_boolean_series]
inverse_filtered_df.view.sort_values()
```

**Conclusion:**  Based on the results of IterativeImputer the original approach of replacing missing values in 'waterfront', 'yr_renovated', and 'view'  with 0 will be taken

#### 4. Replacing NULL values in yr_renovated column with 0. value and changing the type to integer

```python
df.loc[df.yr_renovated.isna(),'yr_renovated']=0.0
df.yr_renovated=df.yr_renovated.astype('int64')
```

#### 5. Replacing '?' values in 'sqft_basement' column with 0. value and changing the type to integer

```python
df.loc[(df.sqft_basement == '?')]
```

```python
# Converting '?' sqft_basement values to 0 and listing unique values
df.loc[(df.sqft_basement == '?'),'sqft_basement']='0.0'

df.sqft_basement=df.sqft_basement.astype('float64')

df.sqft_basement=df.sqft_basement.astype('int64')

#Checking the result
df_temp=df.sqft_basement.unique()
list1 = df_temp.tolist()
list1.sort()
print(list1)
```

#### 6. Replacing NULL values in 'view' column with 0.0 and making it integer

```python
df.loc[df.view.isna(),'view']=0
df.view=df.view.astype('int64')

df.info()
```

# Scrub and Explore

## Additional data cleaning

### Dropping non-needed fields

```python
# Dropping 'zipcode' variable because I think there are better indicators of a location.
# Zipcodes boundaries are usually drawn out of convenience for postal services or
# other more formal reasons than geographic location

# Dropping 'id' field
# Resettingh index because of the removal of the duplicates

df_1=df
df_1=df_1.drop(['id','zipcode'], axis=1)
#df_1.tail()

df_1.reset_index()
```

 **21420** records out of the original **21597** left

## Exploring distributions and correlation of original variables

### Numerical variables: Investigating distributions and correlations between the original, minimally processed predictors and the target (price)

#### Histograms and pair correlations of the original predictor variables

```python
df_1.info()
```

```python
# Investigating histograms

df_1.hist(figsize=(20, 30), bins='auto')
```

**Based on the histograms above**
> 1. The following variables should be considered categorical:
>>Waterfront

Condition

View

> 2. sqft_basement, sqft_lot, sqft_lot15, and yr_renovated have a large number of zeros and are strong candidates for removal of outliers and/or engineered variables

> 3. Latitudes and Longitudes can be used as descriptors of a geographic location of a property. However, I think there is  a better variable to describe the location of a property, a distance from the center of the city, which can be calculated from geocoordinates.

    >4. The target variable, the price of the property, has a strong positive skew attributed to outliers in the higher price bracket. The strategy is to **remove the outliers and to transform the variable** to make it more normally distributed

```python
# Checking for correlations between the variables with Pearson coefficient between 1 and 0.3
# I am using the same approach and reusing the code from Lesson 19

df_coeff=df_1.corr().abs().stack().reset_index().sort_values(0, ascending=False)
df_coeff['pairs'] = list(zip(df_coeff.level_0, df_coeff.level_1))
df_coeff.set_index(['pairs'], inplace = True)
df_coeff.drop(columns=['level_1', 'level_0'], inplace = True)
df_coeff.columns = ['cc']
df_coeff.drop_duplicates(inplace=True)
df_coeff[((df_coeff.cc>.3) & (df_coeff.cc <1))]
```

 **The variables which have the strongest correlations with the price are**

* sqft_living

* grade

* sqft_above

* sqft_living15

* bathrooms

* view

* bedrooms

*lat

#### Establishing an intermediate dataframe and removing some outliers from it

```python
# Removing extreme values from sqft_lot, sqft_lot15, bathrooms, and price
# On one hand, it can be left to the step of outlier removal, on the other doing it now
# will help a visual investigation of the distributions of the variables mentioned above

df_2=df_1[(df_1.sqft_lot >100) & (df_1.sqft_lot<40000)]

df_2=df_2[(df_2.sqft_lot15 >100) & (df_2.sqft_lot15<40000)]

df_2=df_2[df_2.price < 2500000]

df_2=df_2[df_2.bedrooms < 30]

df_2.info()

```

```python
# Creating a new categorical variable: month
df_2['month'] = pd.to_datetime(df_2['date']).dt.month

# Creating a new numerical variable: distance (distance from center)
df_2['distance'] = df_2.apply(lambda row: distance_from_center(row.lat,abs(row.long)), axis=1)

# Creating a new categorical variable: basement_exists (1/0), integer datatype
df_2.loc[(df_2['sqft_basement'] > 50), 'basement_exists'] = 1
df_2.loc[(df_2['sqft_basement'] <= 50), 'basement_exists'] = 0
df_2.basement_exists=df_2.basement_exists.astype('int64')
```

```python
# Creating a new categorical variable (integer datatype) renovation_done with values [0,1,2,3,4]

# 0 representing renovation never done on houses more than 9 years old (yr_built between 2015 and 2006)
# 1 representing renovation done more than or equal 50 years ago
# 2 representing renovation done between 30 and 49 years ago
# 3 representing renovation done between 29 and 10 years ago
# 4 representing renovation done between 9 and 1 year ago OR houses built less or equal 9 years ago
#   (yr_built between 2015 and 2006)

df_2.loc[((df_2['yr_renovated'] == 0) & (df_2['yr_built'] < 2006)), 'renovation_done'] = 0
df_2.loc[((2015-df_2['yr_renovated'] >= 50) & (df_2['yr_renovated'] != 0)), 'renovation_done'] = 1
df_2.loc[((2015-df_2['yr_renovated'] < 50) & (2015-df_2['yr_renovated'] >= 30)), 'renovation_done'] = 2
df_2.loc[((2015-df_2['yr_renovated'] < 30) & (2015-df_2['yr_renovated'] >= 10)), 'renovation_done'] = 3
df_2.loc[((2015-df_2['yr_renovated'] < 10) | (df_2['yr_built'] >= 2006)), 'renovation_done'] = 4

df_2.renovation_done=df_2.renovation_done.astype('int64')
```

```python
# Dropping sqft_basement variable
# Dropping latitude & longtitude variable
# Dropping date variable# Resetting index due to removal of records with extreme values in
# sqft_lot, sqft_lot15, bathrooms and price

df_2=df_2.drop(['sqft_basement','date', 'lat','long','yr_renovated'], axis=1)

df_2=df_2.reset_index(drop=True)

df_2
```

**20015** records out of the original **21597** left

*Index reset*

#### Plotting numerical variables against the target variable

```python
# Dropping categorical variables to simplify the analysis of the numerical ones

df_num1=df_2.drop(['waterfront','view','condition','basement_exists','renovation_done','month'], axis=1)

```

```python
# Displaying the Coefficients of Determination and p-values for the remaining variables
r2_p(df_num1)
```

```python
# Joint plot of the original (not altered) numerical variables
jointplot(df_num1)
```

```python
df_num1.info()
```

```python
# Removing extreme values from bedrooms, bathrooms, floors, and distance
# The observations with the extreme values of these variables are visually identifiable

df_num2=df_num1[(df_num1.bedrooms < 9) & (df_num1.bathrooms < 5.5) &(df_num1.distance < 30)
               & (df_num1.floors<3.5)]
df_num2.drop(['yr_built'], axis=1, inplace=True)
df_num2.reset_index(drop=True, inplace=True)
```

```python
df_num2.info()
```

*df_num2  DataFrame*
**19811** records out of original **21597** left

*Index reset*

```python
# Joint plot of numerical variables after adjustments
jointplot(df_num2)
```

```python
r2_p(df_num2)
```

```python
df_num2.info()
```

#### Visualizing numerical predictors correlation with the price and with each other using heat map

```python
fig, ax = plt.subplots(figsize=(15,10))
sns.heatmap(df_num2.corr(), cmap="cubehelix", annot=True)
```

```python
# Bedrooms is relatively highly correlated with bathrooms and sqft_living; correlations of floors, sqft_lot and sqft_lot15
# with prices are low. Dropping these variables

df_num3=df_num2.drop(['bedrooms','sqft_lot','sqft_lot15','floors'], axis=1)
```

```python
fig, ax = plt.subplots(figsize=(15,10))
sns.heatmap(df_num3.corr(), cmap="cubehelix", annot=True)
```

```python
r2_p(df_num3)
```

#### Exploring the distribution of the remaining numerical predictors

```python
# Plotting KDE and distribution plots of the remaining variables

continuous=['price','sqft_living','sqft_living15','sqft_above','distance']
for col in continuous:
    fig, ax =plt.subplots(figsize=(5, 5))
    distribution(df_num3[col])

discrete=['bathrooms','grade']
for col in discrete:
    fig, ax =plt.subplots(figsize=(5, 5))
    ax = sns.distplot(df_num3[col], bins=10, kde_kws={"color": "r", "lw": 2, "label": "KDE", "bw_adjust": 4})
    skew_val = stats.skew(df_num3[col], bias=False)
    kurt_val = stats.kurtosis(df_num3[col],bias=False)
    print('Column: {} | Skewness = {} | Kurtosis = {}'.
                 format(col, round(skew_val,2), round(kurt_val,2)))
```

*All but one variable (distance) are left shifted, which is indicated by the skewness values >1, with price has the most skewed distribution. Because the skewness it to the left (positive values), the log transformation might be needed to normalize the variables. Kurtosis values for all variables are different from 0 (Pearson's definition of kurtosis of a normal distribution).

    Price distribution is highly Leptokurtic, other variables are slightly Leptokurtic (sqft_living, sqft_above, sqft_living15, grade), slightly Platykurtic (distance) or almost Mesokurtic (bathrooms)*

```python
# Diamond box plots for descrete numerical variables

discrete=['bathrooms','grade']
for col in discrete:

    ax = sns.catplot(x=col, y='price', kind="boxen",
            data=df_num3.sort_values(col), height=9, aspect=12/9)
    ax.set_xticklabels(rotation=45, ha='right', fontsize=12)
    ax.set_yticklabels(fontsize=12)
    plt.ylabel('Price in millions', fontsize=15)
    plt.xlabel(col, fontsize=15);
    plt.grid()
    plt.show()
```

*Based on the distribution plots, variable 'bathroom' has a symmetrical distribution (Skewness is 0.28), with a very low kurtosis (0.09) indicative of a Mesokurtic curve (Gaussian distribution has a kurtosis of 0 by Pearson's definition used by scipy.stats.kurtosis method)

Variable 'grade' is slightly skewed to the right (0.73) and relatively low kurtosis, slightly above 1.

The pronounced correlation of these variables with the price is identifiable in the box plots above. The plots show that the numbers of outliers in the distribution of the variables are reasonable. Both variables have a wider range of values in the higher price brackets.*

### Numerical variables: Exploring Mutual Correlation Coefficients and Variance Inflation Factor

#### Using VIF as an indicator of collinearity between independent variables

```python
calc_vif(df_num3)
```

```python
# Dropping fields that have high VIF. I decided to leave in sqft_living versus sqft_living15 due to the fact that it is
# easier to interpret in a model and it is a feature under control of a property owner (versus sqft_living15 which is
# a charecteristic of a neighborhood)

df_num4=df_num3.drop(['price','sqft_above','sqft_living15'],axis=1)

calc_vif(df_num4)
```

#### Pearson coefficients analysis of the remaining independent variables

```python
df_num4['price']=df_num3['price']
```

```python
# Displaying R2s for the remaining variables
r2_p(df_num4)
```

```python
# Displaying correlations between predictors and the target
jointplot(df_num4)
```

```python
# Using heatmap to visualize mutual relationships between predictors and their correlations with the target
fig, ax = plt.subplots(figsize=(15,10))
sns.heatmap(df_num4.corr(), cmap="Reds", annot=False)
```

```python
# Checking for correlations between the variables with pearson coefficient between 1 and 0.3

df_coeff=df_num4.corr().abs().stack().reset_index().sort_values(0, ascending=False)
df_coeff['pairs'] = list(zip(df_coeff.level_0, df_coeff.level_1))
df_coeff.set_index(['pairs'], inplace = True)
df_coeff.drop(columns=['level_1', 'level_0'], inplace = True)
df_coeff.columns = ['cc']
df_coeff.drop_duplicates(inplace=True)
df_coeff[((df_coeff.cc>.3) & (df_coeff.cc <1))]
```

*Mutual correlation coefficients between the remaining independent variables are
slightly higher or below 0.7. I am leaving sqft_living, grade, and bathroom variables in because of their logical connection with a property price despite their multicollinearity (0.74 & 0.73 are above the 0.7 threshold).*

Therefore the remaining numerical variables for modeling are

1. grade

2. bathrooms

3. sqft_living

4. distance

```python
df_num4.describe()
```

### Categorical variables: Investigating distributions and the raw correlations between the original, minimally processed predictor and the target (price)

#### Original categorical variables: waterfront, view, condition, basement_exists, renovation_done, month

```python
#Creating a DataFrame with categorical variables

df_cat1 = df_2.filter(['price','waterfront','view','condition','basement_exists','renovation_done','month'], axis=1)

```

#### Visual investigation of the box plots

```python
df_cat1
```

```python
names=['waterfront','view','condition','basement_exists','renovation_done','month']
for col in names:
    boxen_plot(df_cat1, col)
```

*Based on the plots, it is quite clear that 'month', 'condition' and 'basement_exists' variables do not affect the price of the properties and can be dropped from the categorical variables*

#### Dumming out variables in the categorical DataFrame

```python
# Using OheHotEncoder to dummy categorical out. Based on the boxplots above, I am leaving
# 'month','basement_exists','condition' variables out, they seem not to have too much effect on the target.
# The reason for removing 'waterfront' variable is a tiny porting of all properties in the dataset.

ohe=OneHotEncoder(drop='first')
df_cat_transform=ohe.fit_transform(df_cat1.drop(['price','month','basement_exists','condition','waterfront'], axis=1))
df_cat1_trsfm=pd.DataFrame(df_cat_transform.todense(),
                          columns=ohe.get_feature_names(['view','renovation_done']))
df_cat1_trsfm.info()
```

```python
# Adding a price column and resetting the index
# Due to the removal of the extreme values in numerical values, some rows have Nan price values; these rows are dropped

df_cat1_trsfm['price']=df_num4['price']

df_cat1_trsfm=df_cat1_trsfm[df_cat1_trsfm.price.notna()]

df_cat1_trsfm=df_cat1_trsfm.reset_index()

df_cat1_trsfm=df_cat1_trsfm.drop('index', axis=1)

df_cat1_trsfm
```

*df_cat1_trsfm DataFrame*
**19811** records out of original **21597** left

**Index reset**

```python
df_cat1_trsfm.info()
```

# Model

## Data Modeling

### Baseline model

>“Everything should be made as simple as possible, but no simpler.”
Albert Einstein

The chosen baseline model is a model with only one numerical variable, grade.

#### Creating a model

```python
## Create the formula and  the model

f  = 'price~grade'

model_baseline = smf.ols(f, df_num4).fit()
model_baseline.summary()
```

```python
fig = sm.graphics.qqplot(model_baseline.resid,dist=stats.norm,fit=True,line='45')
```

*The baseline model has a coefficient of determination of 0.452, indicating that roughly 45% of the observations fit the model. F-statistics is very high that indicates that the baseline model is a significant improvement of the  "intercept only model"

    The Skewness and the Kurtosis values indicate non-normal distribution of the target variable

    QQ plot is also indicative of the abnormal distribution of the residuals, especially in the upper Quantile*

### Model 1 (all numerical variables considered significant, see Explore section)

```python
## Create a formula including the remaining numerical variables

variables_to_include = ' + '.join(df_num4.drop('price',axis=1).columns)

## Create the formula and  the model
f  = "price~" + variables_to_include

model_1 = smf.ols(f, df_num4).fit()
model_1.summary()
```

#### QQ plot: to access normality of the target variable

```python
fig = sm.graphics.qqplot(model_1.resid,dist=stats.norm,fit=True,line='45')
```

*The summary of the model above indicates that

    1. All the independent variables coefficients and the intercept value are significant (p-values< 0.05)

    2. The coefficient of determination (R2) is not very high, but it is significantly higher than R2 of the baseline model. It indicates that about 66.8 percent of the observations fall within the regression line

    3. The skew and the Kurtosis values indicate the highly non-normal distribution of the target variable

    4. The high value of JB coefficient also indicates that the data is highly non-normal
*

*From the model's QQ plot, it is also quite obvious that the 'price' target variable is not normally distributed. A steep swing up indicates that the higher-priced houses are less likely to fit the baseline model and are more spread out. One possible reason might be an unusually large number of outliers in the dataset*

*There are two potential approaches that can be taken

        1. Normalization by either log or square root transformation

        2. Removal of outliers*

####  Plot regression results against each regressor: accessing linearity of there relationship with the target and their homoscedasticity

```python
for col in (df_num4.drop('price',axis=1).columns):
    fig = sm.graphics.plot_regress_exog(model_1, col, fig=plt.figure(figsize=(12,8)))
```

*The results indicate that all of the predictors display linear relationship with the target. The distance, sqft_living and bathrooms variables display less  heteroscedasticity than the grade variable does.

There might be several appropriate ways to address this issue

    1. Log transformation of the target and/or independent variables

    2. Using either Generalized Least Squares or Weighted Least squares

    3. Bootstrapping
*

### Model 2 (adding categorical variables)

#### Creating a model with all numerical variables and all categorical variables

```python
df_num4
```

```python
df_cat1_trsfm
```

```python
# DatyaFrames to concatinate: df_cat1_trsfm AND df_num4
df_num_cat_1 = pd.concat([df_num4, df_cat1_trsfm.drop('price',axis=1)],axis=1)
```

```python
# Visualizing multicollinearity in the new dataset
fig, ax = plt.subplots(figsize=(15,10))
sns.heatmap(df_num_cat_1.corr(), cmap="cubehelix", annot=True)
```

*The matrix indicates no strong correlation between the price and any of the categorical variables. However, renovation_done_4 correlation is slightly higher than the rest of the categorical variables. Conceptually this variables indicative of a recent renovation or a newer property.

There is no high expectations that adding the categorical variables to the mix will significantly improve the model.*

```python
df_num_cat_1
```

```python
# Create a formula for the numerical variables from the basemodel
# AND the categorical variables from the previous section

variables_to_include = ' + '.join(df_num_cat_1.drop('price',axis=1).columns)

## Create the formula and the model
f  = "price~" + variables_to_include

model_2 = smf.ols(f, df_num_cat_1).fit()
model_2.summary()
```

```python
fig = sm.graphics.qqplot(model_2.resid,dist=stats.norm,fit=True,line='45')
```

*The summary indicates a very slight improvement over the previous model, 66.9% versus 66.8% of all of the observations fall within the results of the line formed by the regression equation.

    It is also evident that p-values of most of the categorical values are very high, indicating their insignificance in the model. However, because they describe the same feature, I am leaving them in for now

The residuals normality did not improve*

### Model 3 (preprocessing and removal of outliers)

#### Scaling with Robust Scaler

*I ruled out the scaling of the data in this step of the process because it would not change the statistics of the mode, though the correlation coefficients would become more compatible with each other.

I am leaving the snippets in the notebook in case I reconsider.

The next step is a removal of ourliers*

```python
# Using RobustScaler to scale the data
"""# Using RobustScaler, which transforms the feature vector by subtracting the median and then dividing by the
# interquartile range (%25-75%). It is the most robust to the ourliers

df_for_scalers=df_num_cat_2.copy()

cols=['bathrooms','grade','sqft_living15','distance','waterfront_1','renovation_done_4']
scaler = RobustScaler()
robust_df = scaler.fit_transform(df_for_scalers.drop('price',axis=1))
robust_df = pd.DataFrame(robust_df, columns=cols)

df_num_cat_2.describe()"""

"""robust_df.describe()"""

"""fig, (ax1, ax2) = plt.subplots(ncols = 2, figsize =(20, 5))
ax1.set_title('Before Scaling')

sns.kdeplot(df_for_scalers['distance'], ax = ax1, color ='blue')

ax2.set_title('After Robust Scaling')

sns.kdeplot(robust_df['distance'], ax = ax2, color ='red')

plt.show()"""
```

```python
# Building and summary of the model with scaled data
"""robust_df['price']=df_for_scalers['price']
robust_df.info()

num_cat_var2_robust = ' + '.join(robust_df.drop('price',axis=1).columns)

## Create the formula and the model
f  = "price~" + num_cat_var2_robust

model_num_cat_2_robust = smf.ols(f, robust_df).fit()
model_num_cat_2_robust.summary()

coeffs=model_num_cat_2.params
coeffs.sort_values().round(2)

coeffs=model_num_cat_2_robust.params
coeffs.sort_values().round(2)"""
```

#### Removal of outliers

##### IQR method

###### Using IQR

```python
df_num_cat_1
```

```python
df_num_cat_2=df_num_cat_1.copy()

# Due to the fact that discrete variables are not suitable for IQR method outlier removal, they are being dropped
# from the DataFrame. They will be added back to the Dataframe for modeling

df_num_cat_2=df_num_cat_2.drop(['view_1','view_2','view_3','view_4',
                              'renovation_done_1','renovation_done_2','renovation_done_3','renovation_done_4',
                              'grade','bathrooms'], axis=1)

Q1 = df_num_cat_2.quantile(q=.25)
Q3 = df_num_cat_2.quantile(q=.75)
IQR = df_num_cat_2.apply(stats.iqr)

df_num_cat_3 = df_num_cat_2[~((df_num_cat_2 < (Q1-1.5*IQR)) | (df_num_cat_2 > (Q3+1.5*IQR))).any(axis=1)]

df_num_cat_3
```

```python
df_num_cat_3['grade']=df_num_cat_1['grade']
df_num_cat_3['bathrooms']=df_num_cat_1['bathrooms']
df_num_cat_3['view_1']=df_num_cat_1['view_1']
df_num_cat_3['view_2']=df_num_cat_1['view_2']
df_num_cat_3['view_3']=df_num_cat_1['view_3']
df_num_cat_3['view_4']=df_num_cat_1['view_4']
df_num_cat_3['renovation_done_1']=df_num_cat_1['renovation_done_1']
df_num_cat_3['renovation_done_2']=df_num_cat_1['renovation_done_2']
df_num_cat_3['renovation_done_3']=df_num_cat_1['renovation_done_3']
df_num_cat_3['renovation_done_4']=df_num_cat_1['renovation_done_4']
```

```python
df_num_cat_3.info()
```

```python
df_num_cat_3=df_num_cat_3.reset_index()

df_num_cat_3.info()
```

```python
df_num_cat_3=df_num_cat_3.drop('index', axis=1)
df_num_cat_3.info()
```

*df_num_cat_3 DataFrame

18619 records out of the original 21597 left

Index reset*

###### Building the model

```python
## Formula is the same, model is for the cleaned DF

variables_to_include_3_1 = ' + '.join(df_num_cat_3.drop('price',axis=1).columns)
f  = "price~" + variables_to_include_3_1

model_3_1 = smf.ols(f, df_num_cat_3).fit()
model_3_1.summary()
```

```python
fig = sm.graphics.qqplot(model_3_1.resid,dist=stats.norm,fit=True,line='45')
```

*While the IQR removal of outliers decreased the R squared of the model, it made the distribution more normal (Skew and Kurtosis values are almost within the normality ranges). This fact is also reflected by the QQ plot of the model residuals. Unfortunately, the Coefficient of determination dropped*

##### Z-score method

###### Using Z scores

```python
df_num_cat_1
```

```python
df_num_cat_4=df_num_cat_1.copy()

df_num_cat_4=df_num_cat_4.drop(['view_1','view_2','view_3','view_4',
                              'renovation_done_1','renovation_done_2','renovation_done_3','renovation_done_4',
                              'grade','bathrooms'], axis=1)

df_num_cat_4['z_sqft_living']=stats.zscore(df_num_cat_4['sqft_living'])
df_num_cat_4['z_distance']=stats.zscore(df_num_cat_4['distance'])
df_num_cat_4['z_price']=stats.zscore(df_num_cat_4['price'])

df_num_cat_4
```

```python
df_num_cat_5=df_num_cat_4[(abs(df_num_cat_4.z_price) < 3)]

df_num_cat_5=df_num_cat_5[(abs(df_num_cat_5.z_distance) < 3)]

df_num_cat_5=df_num_cat_5[(abs(df_num_cat_5.z_sqft_living) < 3)]
df_num_cat_5
```

```python
df_num_cat_5['grade']=df_num_cat_1['grade']
df_num_cat_5['bathrooms']=df_num_cat_1['bathrooms']
df_num_cat_5['view_1']=df_num_cat_1['view_1']
df_num_cat_5['view_2']=df_num_cat_1['view_2']
df_num_cat_5['view_3']=df_num_cat_1['view_3']
df_num_cat_5['view_4']=df_num_cat_1['view_4']
df_num_cat_5['renovation_done_1']=df_num_cat_1['renovation_done_1']
df_num_cat_5['renovation_done_2']=df_num_cat_1['renovation_done_2']
df_num_cat_5['renovation_done_3']=df_num_cat_1['renovation_done_3']
df_num_cat_5['renovation_done_4']=df_num_cat_1['renovation_done_4']
```

```python
df_num_cat_5=df_num_cat_5.drop(['z_sqft_living','z_distance','z_price'], axis=1)
```

```python
df_num_cat_5=df_num_cat_5.reset_index()
df_num_cat_5=df_num_cat_5.drop('index', axis=1)
df_num_cat_5.info()
```

*df_num_cat_5 DataFrame

19261 records out of the original 21597 left

Index reset*

##### Building the model

```python
## Formula is the same, model is for the cleaned DF

variables_to_include_3_2 = ' + '.join(df_num_cat_5.drop('price',axis=1).columns)
f  = "price~" + variables_to_include_3_2

model_3_2 = smf.ols(f, df_num_cat_5).fit()
model_3_2.summary()
```

```python
fig = sm.graphics.qqplot(model_3_2.resid,dist=stats.norm,fit=True,line='45')
```

*The R squared of the model is 0.654 and F-statistics is higher than for the previous model

The IQR method of outliers removal made the residual distribution more normal than Z-score method due to the former having more strict criteria. The decision is to use the dataset compiled after Z-score outlier removal.

The next step is Log transformation of the target variable*

### Model 4 (Using log and square root transformations on the target variable)

#### Log Transformation

```python
df_num_cat_5.info()
```

```python
# Log transform
df_num_cat_5_log=df_num_cat_5.copy()

df_num_cat_5_log['log_price'] = df_num_cat_5['price'].map(lambda x: np.log(x))
df_num_cat_5_log
```

```python
df_num_cat_5_log.info()
```

```python
# Histogram of log_price and price
continuous=['price','log_price']
for col in continuous:
    fig, ax =plt.subplots(figsize=(5, 5))
    distribution(df_num_cat_5_log[col])

```

*The transformation worked well, improving the normality of the 'price' variable. Log_price distribution looks more symmetrical. Skewness improved dramatically (from 1.18 to -0.01, 0 being perfectly symmetrical)

 Kurtosis value decreased, making the curve more Mesokurtic (close to a Gaussian curve). It is an expected effect of log transform.

The next step is test a square root transformation.*

```python
# Square transform
df_num_cat_5_sqrt=df_num_cat_5.copy()

df_num_cat_5_sqrt['sqrt_price'] = df_num_cat_5['price'].map(lambda x: np.sqrt(x))
df_num_cat_5_sqrt
```

```python
# Histogram of log_price and price
continuous=['price','sqrt_price']
for col in continuous:
    fig, ax =plt.subplots(figsize=(5, 5))
    distribution(df_num_cat_5_sqrt[col])

```

*The square root transformation also worked well in improving the normality of the 'price' variable.
sqrt_price distribution looks more symmetrical. Skewness improved dramatically (from 1.18 to -0.59, 0 being perfectly symmetrical)

 However, both of the parameters are worse that the parameters of log_price distribution.

The next step is create two separate models and to see if the transformations made a difference*

#### Model using log transformed target variable

```python
## Formula is the same, model is for the cleaned DF

variables_to_include_4_1 = ' + '.join(df_num_cat_5_log.drop(['price','log_price'],axis=1).columns)
f  = "log_price~" + variables_to_include_4_1

model_4_1 = smf.ols(f, df_num_cat_5_log).fit()
model_4_1.summary()
```

#### Model using square root transformed target variable

```python
## Formula is the same, model is for the cleaned DF

variables_to_include_4_2 = ' + '.join(df_num_cat_5_sqrt.drop(['price','sqrt_price'],axis=1).columns)
f  = "sqrt_price~" + variables_to_include_4_2

model_4_2= smf.ols(f, df_num_cat_5_sqrt).fit()
model_4_2.summary()
```

```python
fig = sm.graphics.qqplot(model_4_1.resid,dist=stats.norm,fit=True,line='45')
```

```python
fig = sm.graphics.qqplot(model_4_2.resid,dist=stats.norm,fit=True,line='45')
```

*Both models have improved the R squared and the F-statistics of the previous models. The residuals of both models display a close-to-normal distribution. Log transformation helped improve the upper part of the distribution, while square root transformation worked better in the lower part of the distribution.

R squared of the square root transformation-based model is slightly higher, while its kurtosis value is slightly worse than the kurtosis value of the log-transformed price model. The decision is to use the log-transformed target variable.

The next step is to remove unnecessary categorical variables, scale the remaining variables, and built the last model with coefficients in the regression model, which are easy to compare and interpret*

```python
# Removing all categorical values with high p-values and renaming the renovation_done_4
# to a more prominent name, reflective of the definition of the field (see Categorical variables subsection
# in Exploring distributions and correlations section)

df_num_cat_5_log_cropped=df_num_cat_5_log.copy()
df_num_cat_5_log_cropped=df_num_cat_5_log_cropped.drop(['view_1','view_2','view_3','view_4','renovation_done_1',
                                                'renovation_done_2','renovation_done_3'], axis=1)
df_num_cat_5_log_cropped.rename(columns={'renovation_done_4': 'recent_renovation_new'}, inplace=True)
```

```python
df_num_cat_5_log_cropped
```

```python
# Standardizing the independent variables. I decided to do it manually due to a better control of the utput

df_num_cat_5_log_cropped_st=df_num_cat_5_log_cropped.copy()

lv_min=df_num_cat_5_log_cropped_st.sqft_living.min()
lv_range=df_num_cat_5_log_cropped_st.sqft_living.max()-df_num_cat_5_log_cropped_st.sqft_living.min()
distance_range=df_num_cat_5_log_cropped_st.distance.max()-df_num_cat_5_log_cropped_st.distance.min()
distance_min=df_num_cat_5_log_cropped_st.distance.min()
bathroom_range=df_num_cat_5_log_cropped_st.bathrooms.max()-df_num_cat_5_log_cropped_st.bathrooms.min()
bathroom_min=df_num_cat_5_log_cropped_st.bathrooms.min()
grade_range=df_num_cat_5_log_cropped_st.grade.max()-df_num_cat_5_log_cropped_st.grade.min()
grade_min=df_num_cat_5_log_cropped_st.grade.min()

df_num_cat_5_log_cropped_st['sqft_living_st']=df_num_cat_5_log_cropped_st.apply(lambda row: round((row.sqft_living-lv_min)/lv_range,3), axis=1)
df_num_cat_5_log_cropped_st['distance_st']=df_num_cat_5_log_cropped_st.apply(lambda row: round((row.distance-distance_min)/distance_range,3), axis=1)
df_num_cat_5_log_cropped_st['bathrooms_st']=df_num_cat_5_log_cropped_st.apply(lambda row: round((row.bathrooms-bathroom_min)/bathroom_range,3), axis=1)
df_num_cat_5_log_cropped_st['grade_st']=df_num_cat_5_log_cropped_st.apply(lambda row: (row.grade-grade_min)/grade_range, axis=1)
#df_num_cat_5_sqrt_cropped_st['recent_renovation_new_str']=df_num_cat_5_sqrt_cropped_st['recent_renovation_new'].astype('str')

df_num_cat_5_log_cropped_st
```

```python
## Formula is the same, model is for the cleaned DF

variables_to_include_4_3 = ' + '.join(df_num_cat_5_log_cropped_st.drop(
    ['price','log_price','sqft_living','distance','grade','bathrooms'],axis=1).columns)
f  = "log_price~" + variables_to_include_4_3
print(f)
model_4_3= smf.ols(f, df_num_cat_5_log_cropped_st).fit()
model_4_3.summary()
```

```python
fig = sm.graphics.qqplot(model_4_3.resid,dist=stats.norm,fit=True,line='45')

# Removing variable to simplify the display (categorical variable with only 2 values does not have
# much use when plotted by regress_exog)
# print(df_num_cat_5_log_cropped_st.drop(['price','log_price','recent_renovation_new',
#                                        'sqft_living', 'distance', 'grade', 'bathrooms'],axis=1).columns)

for col in (df_num_cat_5_log_cropped_st.drop(['price','log_price','recent_renovation_new',
                                        'sqft_living', 'distance', 'grade', 'bathrooms'],axis=1).columns):
    fig = sm.graphics.plot_regress_exog(model_4_3, col, fig=plt.figure(figsize=(12,8)))
```

*The final linear regression model of a log of the price variable versus grade, bathrooms, distance from the center of the city, sqft_living space, and the indicator if a house has been renovated recently or a newer house has a Coefficient of Determination of 0.663. It is indicative of the fact that 66.3% of the sold properties fall within the results of the line formed by the regression equation. F-statistics displays the high value and the overall p-value much lower than the confidence interval

The independent variables used in the equation display a clear linear relationship with the target and homoscedasticity.

The next step is to validate the model by using training and test datasets*

### Train the model

```python
# Define X and y
y=df_num_cat_5_log_cropped_st[['log_price']]
X=df_num_cat_5_log_cropped_st.drop(['price','log_price','recent_renovation_new',
                                        'sqft_living', 'distance', 'grade', 'bathrooms'], axis=1)
```

```python
y
```

```python
X
```

```python
# Split the data into training and test sets. Use the default split size
# X_train, X_test, y_train, y_test = train_test_split(X, y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=100)

print(len(X_train), len(X_test), len(y_train), len(y_test))
```

```python
# Fit the model to train data
linreg = LinearRegression()
model=linreg.fit(X_train, y_train)

# Calculate predictions on training and test sets for y_hat
y_hat_train=linreg.predict(X_train)
y_hat_test=linreg.predict(X_test)

# Calculate residuals
train_residuals=y_hat_train-y_train
test_residuals=y_hat_test-y_test

# Calculate training and test RMSE
train_mse = mean_squared_error(y_train, y_hat_train, squared=False)
test_mse = mean_squared_error(y_test, y_hat_test, squared=False)
print('Train MSE:', round(train_mse,3))
print('Test MSE:', round(test_mse,3))
```

```python
print(model.coef_, model.intercept_, model.score(X_test, y_test))
```

### Validation

```python
price_target=df_num_cat_5_log_cropped_st[['log_price']]
price_predictors= df_num_cat_5_log_cropped_st.drop(['price','log_price','recent_renovation_new',
                                        'sqft_living', 'distance', 'grade', 'bathrooms'], axis=1)
```

```python
metrics.r2_score(price_target, linreg.predict(price_predictors))
```

```python
metrics.mean_absolute_error(price_target, linreg.predict(price_predictors))
```

```python
mean_squared_error(price_target, linreg.predict(price_predictors))
```

>Train MSE: 0.272

Test MSE: 0.272

Are equal down to the third decimal digit indicating a good agreement between the training and the test sets

R squared for for the prediction on the full dataset is the same as in model_4_3: 0.663

Mean Absolute Error for the prediction on the full dataset is 0.213 which is not great but acceptable. The best possible theoretical value is 0.

Mean Squared Error for the prediction on the full dataset is 0.074. The best possible theoretical value is 0.

```python
# Not sure if the results of this function are indicative of anything, but leaving it here for a possible future use
"""mse, bias, var = bias_variance_decomp(model, X_train.values, y_train.values.flatten(),
                                      X_test.values, y_test.values, num_rounds=200, random_seed=1, loss='mse')
# summarize results
print('MSE: %.3f' % mse)
print('Bias: %.3f' % bias)
print('Variance: %.3f' % var)"""
```

# iNterpret

```latex
%%latex

\begin{align}\ln\left( {Price} \right) = 12.366 + 1.265 \cdot \left( grade \right) +
1.080 \cdot \left( sqft\_living \right) - 0.989 \cdot \left( distance \right)
+ 0.060 \cdot \left( bathrooms \right) - 0.035 \cdot \left( recent\_renovation\_new \right) \end{align}
```

The final model has a reasonable predictive ability tested in the final step of the model validation. MSE, MAE, and R2 score along with the model p-values for all predictors indicate a good fit.

The most influential predictor is a **building grade**, following by a **living space footage**. Both factors are **positively correlated** with the price of the property. Both factors are within property owners' control when they are renovating their houses.
A **distance** from the center of the city is **negatively correlated** with the price of property, meaning the further away a property is, the lower is the price. It is not a controllable variable but is helpful for home buyers if the living space and the number of bedrooms/bathrooms are important.
 A **number of bathrooms** has a **positive effect** on the price of a property, but it is not as strong as the first two factors. This fact indicates that the convenience of having multiple bathrooms is essential for potential buyers and should be taken into account when owners are planning a renovation.

The last predictor in the model is an indicator of whether a property **has been renovated recently or a new construction**. It is very **weakly negatively correlated** with the price variable. The negative correlation (reduction of the price) might be related to the following factors: newer properties, on the average, are of less building quality.

**The intercept** of the model is a **bias** of the model and can be interpreted as an offset of the model due to other factors not taken into account for various reasons.

# Conclusions and Recommendation

**Recommendations to property owners planning a renovation to their properties:**
* Increase the living space of your property
* Do the renovation with higher building quality
* Consider adding a bathroom

**Recommendations to potential buyers:**
* Look for properties further away from the city center to make the best out of your property buying budget
* Properties in some zipcodes of the city are more affordable than others at the same distance from the city center
* Properties in some zipcodes of the city are more affordable than others with a better view, more considerable property lots, and with older houses of better quality construction if these factors are essential to a buyer

**Limitations of the model:**
* The original dataset does not include other important factors, and therefore the model is biased
* Multiple linear regression models, while easily interpretable, are limited in their predictive ability
* Some variables in the dataset are strongly correlated with each other, and that affect the predictive power of the model

**Suggestion for future improvements**:
* Add variables to the original dataset like kitchen renovation, average commute time, crime index, average nearby public school quality, etc.
* Update the dataset with more current data

# Appendix

>Exporting DataFrames to CSV files

```python
df_test=X_test.copy()
df_test['log_price']=y_test['log_price']
df_test['recent_renovation_new']=df_num_cat_5_log_cropped_st['recent_renovation_new']
df_test['sqft_living']=df_num_cat_5_log_cropped_st['sqft_living']
df_test['distance']=df_num_cat_5_log_cropped_st['distance']
df_test['grade']=df_num_cat_5_log_cropped_st['grade']
df_test['bathrooms']=df_num_cat_5_log_cropped_st['bathrooms']
df_test['price']=df_test.apply(lambda row: math.exp(row.log_price), axis=1)
df_test['recent_renovation_new_str']=df_test['recent_renovation_new'].astype('str')
#df_test=df_test.reset_index(drop='index')
df_test
```

```python
df_test.to_csv('data/df_test.csv', encoding='utf-8')
```

```python
df_zipcode_viz=df.groupby('zipcode').mean()

df_zipcode_viz=df_zipcode_viz.reset_index()

df_zipcode_viz=df_zipcode_viz.drop(['id','sqft_above','sqft_basement','yr_renovated','lat','long'], axis=1)
df_zipcode_viz
```

```python
df_zipcode_viz.to_csv('data/df_zipcode_vs.csv', encoding='utf-8')
```

> Link to Visualization Notebook

[Visualization notebook](./visualization_appendix.ipynb)