# 如何對資料框進行基礎操作

> 寫一些 pandas 語法操作 Johns Hopkins COVID-19 每日報告
>
> 標籤：程式設計，獲取載入，整併轉換

郭耀仁 <yaojenkuo@datainpoint.com>

```python
# 載入專案需要使用的套件
import datetime
import pandas as pd
```

## TL; DR

在這個專案中，我們打算寫一些 pandas 語法操作約翰霍普金斯大學 COVID-19 Data Repository 中最新的每日報告，讀者將學會如何定義函式 `get_latest_daily_report()` 將約翰霍普金斯大學 COVID-19 Data Repository 中最新的每日報告載入成為資料框、如何衍生新的欄位（治療中案例數）、選擇特定欄位（挑出一個或多個）、篩選指定觀測值（台灣在哪裡）、分組摘要（以國家為單位聚合確診數）以及排序（將摘要結果由大到小遞減排序）。

## 資料來源

資料來源是 [COVID-19 Data Repository by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University](https://github.com/CSSEGISandData/COVID-19) 中的每日報告資料夾 `/csse_covid_19_data/csse_covid_19_daily_reports`，該資料夾從 2020-01-22 開始每天都有一個單獨的 CSV 檔案記錄該日的全球現況。

## 載入最新的每日報告

CSV 檔案的命名是以 `%m-%d-%Y` Unix 時間格式、俗稱的 `mm-dd-yyyy` 格式作為檔案名稱，如果希望載入最新的每日報告，我們可以用電腦的當天日期作為檔名，但是由於資料源更新時間、時區的差異，使用當天日期很有可能沒有對應的檔案，因此我們可以寫一段程式，他的處理邏輯是：

1. 先以電腦的當天日期作為檔名是否可以載入成功
2. 如果成功這段程式的任務就完成了
3. 如果載入失敗產生錯誤訊息，就將當天日期減去 1，直到載入成功

這段程式需要 Python 的標準套件 `datetime`、第三方套件 `pandas`、`try...except...` 語法以及 `while` 語法。其中 `datetime` 可以協助我們獲得電腦的當天日期、進行日期的運算以及調整日期的文字格式。

```python
latest_date = datetime.date.today()
latest_date_fmt = latest_date.strftime('%m-%d-%Y')
print(latest_date_fmt)
```

    09-06-2020

第三方套件 `pandas` 可以協助我們將 CSV 檔案讀入成為方便摘要分析的 `DataFrame`，每日報告由於更新時間、時區差異的緣故，大概都會是昨天或者前天，因此如果貿然將當天日期作為檔名通常會獲得錯誤訊息。這時我們就可以利用 `try...except...` 將錯誤捕捉起來。

```python
csv_url = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/{}.csv".format(latest_date_fmt)
try:
    daily_report = pd.read_csv(csv_url)
except:
    print("尚未有 {} 的每日報告。".format(latest_date_fmt))
```

    尚未有 09-06-2020 的每日報告。

最後加入 `while` 語法，目的是只要錯誤被捕捉起來，就將當天日期減 1 成為再前一天日期，再嘗試一次載入，假若再有錯誤被捕捉，就持續減去 1 天，直到成功為止。

```python
latest_date = datetime.date.today()
day_delta = datetime.timedelta(days=1)
fmt = '%m-%d-%Y'
while True:
    try:
        latest_date_fmt = latest_date.strftime(fmt)
        csv_url = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/{}.csv".format(latest_date_fmt)
        daily_report = pd.read_csv(csv_url)
        print("載入了 {} 的每日報告。".format(latest_date_fmt))
        break
    except:
        latest_date_fmt = latest_date.strftime(fmt)
        print("尚未有 {} 的每日報告。".format(latest_date_fmt))
        latest_date -= day_delta
```

    尚未有 09-06-2020 的每日報告。
    載入了 09-05-2020 的每日報告。

## 將載入最新的每日報告包裝成函式

將前面的程式包裝成函式，可以回傳最新每日報告以及檔案日期。

```python
def get_latest_daily_report():
    """
    This function returns the latest global daily report from https://github.com/CSSEGISandData/COVID-19 and its file date.
    """
    latest_date = datetime.date.today()
    day_delta = datetime.timedelta(days=1)
    fmt = '%m-%d-%Y'
    while True:
        try:
            latest_date_fmt = latest_date.strftime(fmt)
            csv_url = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/{}.csv".format(latest_date_fmt)
            daily_report = pd.read_csv(csv_url)
            print("載入了 {} 的每日報告。".format(latest_date_fmt))
            break
        except:
            latest_date_fmt = latest_date.strftime(fmt)
            print("尚未有 {} 的每日報告。".format(latest_date_fmt))
            latest_date -= day_delta
    return latest_date, daily_report
```

```python
latest_date, daily_report = get_latest_daily_report()
```

    尚未有 09-06-2020 的每日報告。
    載入了 09-05-2020 的每日報告。

```python
daily_report.shape # 每日報告的外觀
```

    (3954, 14)

```python
daily_report.head() # 每日報告的前五列
```

      FIPS
      Admin2
      Province_State
      Country_Region
      Last_Update
      Lat
      Long_
      Confirmed
      Deaths
      Recovered
      Active
      Combined_Key
      Incidence_Rate
      Case-Fatality_Ratio

      0
      NaN
      NaN
      NaN
      Afghanistan
      2020-09-06 04:28:33
      33.93911
      67.709953
      38324
      1409
      30082
      6833.0
      Afghanistan
      98.447555
      3.676547

      1
      NaN
      NaN
      NaN
      Albania
      2020-09-06 04:28:33
      41.15330
      20.168300
      10102
      312
      5976
      3814.0
      Albania
      351.032038
      3.088497

      2
      NaN
      NaN
      NaN
      Algeria
      2020-09-06 04:28:33
      28.03390
      1.659600
      46071
      1549
      32481
      12041.0
      Algeria
      105.062495
      3.362202

      3
      NaN
      NaN
      NaN
      Andorra
      2020-09-06 04:28:33
      42.50630
      1.521800
      1215
      53
      928
      234.0
      Andorra
      1572.510192
      4.362140

      4
      NaN
      NaN
      NaN
      Angola
      2020-09-06 04:28:33
      -11.20270
      17.873900
      2935
      117
      1192
      1626.0
      Angola
      8.930129
      3.986371

## 衍生新的欄位

治療中案例數可以用既有的欄位計算而得，常見的定義為：確診數減去死亡數再減痊癒數。

\begin{equation}
\text{Active} = \text{Confirmed} - \text{Deaths} - \text{Recovered}
\end{equation}

```python
active = daily_report["Confirmed"] - daily_report["Deaths"] - daily_report["Recovered"]
active
```

    0        6833
    1        3814
    2       12041
    3         234
    4        1626
            ...
    3949     8737
    3950        1
    3951      214
    3952      749
    3953     1286
    Length: 3954, dtype: int64

與原本資料框中的欄位 Active 比對可以檢查每日報告中有問題的觀測值紀錄。

```python
print((active != daily_report["Active"]).sum()) # 衍生欄位與原有欄位 Active 不同的觀測值數有幾筆
daily_report[active != daily_report["Active"]]  # 問題的觀測值紀錄
```

    4

      FIPS
      Admin2
      Province_State
      Country_Region
      Last_Update
      Lat
      Long_
      Confirmed
      Deaths
      Recovered
      Active
      Combined_Key
      Incidence_Rate
      Case-Fatality_Ratio

      67
      NaN
      NaN
      Diamond Princess
      Canada
      2020-09-06 04:28:33
      NaN
      NaN
      0
      1
      0
      NaN
      Diamond Princess, Canada
      NaN
      NaN

      96
      NaN
      NaN
      Unknown
      Chile
      2020-09-06 04:28:33
      NaN
      NaN
      54
      1
      54
      NaN
      Unknown, Chile
      NaN
      1.851852

      740
      90004.0
      Unassigned
      Arizona
      US
      2020-09-06 04:28:33
      NaN
      NaN
      1
      1
      0
      NaN
      Unassigned, Arizona, US
      NaN
      100.000000

      1517
      19159.0
      Ringgold
      Iowa
      US
      2020-09-06 04:28:33
      40.735189
      -94.243685
      34
      1
      0
      32.0
      Ringgold, Iowa, US
      694.728239
      2.941176

在資料框的第 0 欄插入衍生欄位。

```python
daily_report.insert(0, 'Derived_Active', active)
daily_report.head()
```

      Derived_Active
      FIPS
      Admin2
      Province_State
      Country_Region
      Last_Update
      Lat
      Long_
      Confirmed
      Deaths
      Recovered
      Active
      Combined_Key
      Incidence_Rate
      Case-Fatality_Ratio

      0
      6833
      NaN
      NaN
      NaN
      Afghanistan
      2020-09-06 04:28:33
      33.93911
      67.709953
      38324
      1409
      30082
      6833.0
      Afghanistan
      98.447555
      3.676547

      1
      3814
      NaN
      NaN
      NaN
      Albania
      2020-09-06 04:28:33
      41.15330
      20.168300
      10102
      312
      5976
      3814.0
      Albania
      351.032038
      3.088497

      2
      12041
      NaN
      NaN
      NaN
      Algeria
      2020-09-06 04:28:33
      28.03390
      1.659600
      46071
      1549
      32481
      12041.0
      Algeria
      105.062495
      3.362202

      3
      234
      NaN
      NaN
      NaN
      Andorra
      2020-09-06 04:28:33
      42.50630
      1.521800
      1215
      53
      928
      234.0
      Andorra
      1572.510192
      4.362140

      4
      1626
      NaN
      NaN
      NaN
      Angola
      2020-09-06 04:28:33
      -11.20270
      17.873900
      2935
      117
      1192
      1626.0
      Angola
      8.930129
      3.986371

## 選擇特定欄位

在中括號裡頭輸入欄位名稱可以將資料以 `Series` 外型從資料框中取出。

```python
daily_report["Country_Region"]
```

    0              Afghanistan
    1                  Albania
    2                  Algeria
    3                  Andorra
    4                   Angola
                   ...
    3949    West Bank and Gaza
    3950        Western Sahara
    3951                 Yemen
    3952                Zambia
    3953              Zimbabwe
    Name: Country_Region, Length: 3954, dtype: object

若想要選擇多個欄位，將多個欄位名稱以 `list` 傳入中括號。

```python
multiple_columns = ["Country_Region", "Confirmed", "Deaths", "Recovered"]
daily_report[multiple_columns]
```

      Country_Region
      Confirmed
      Deaths
      Recovered

      0
      Afghanistan
      38324
      1409
      30082

      1
      Albania
      10102
      312
      5976

      2
      Algeria
      46071
      1549
      32481

      3
      Andorra
      1215
      53
      928

      4
      Angola
      2935
      117
      1192

      ...
      ...
      ...
      ...
      ...

      3949
      West Bank and Gaza
      25575
      177
      16661

      3950
      Western Sahara
      10
      1
      8

      3951
      Yemen
      1983
      572
      1197

      3952
      Zambia
      12709
      292
      11668

      3953
      Zimbabwe
      6837
      206
      5345

3954 rows × 4 columns

## 篩選指定觀測值

在中括號裡頭輸入判斷條件所獲得的布林值 `Series` 可以獲得指定的觀測值。

```python
is_tw = daily_report['Country_Region'] == 'Taiwan*' # 台灣在哪裡
is_tw
```

    0       False
    1       False
    2       False
    3       False
    4       False
            ...
    3949    False
    3950    False
    3951    False
    3952    False
    3953    False
    Name: Country_Region, Length: 3954, dtype: bool

```python
daily_report[is_tw] # is_tw 是一個布林值 Series
```

      Derived_Active
      FIPS
      Admin2
      Province_State
      Country_Region
      Last_Update
      Lat
      Long_
      Confirmed
      Deaths
      Recovered
      Active
      Combined_Key
      Incidence_Rate
      Case-Fatality_Ratio

      622
      12
      NaN
      NaN
      NaN
      Taiwan*
      2020-09-06 04:28:33
      23.7
      121.0
      492
      7
      473
      12.0
      Taiwan*
      2.065771
      1.422764

## 分組摘要

以國家為單位，所以在 `groupby()` 方法中傳入 Country_Region 欄位，獲得一個 `DataFrameGroupBy` 類別。

```python
daily_report.groupby("Country_Region")
```

    <pandas.core.groupby.generic.DataFrameGroupBy object at 0x7ffe6f5f12b0>

指定 `DataFrameGroupBy` 類別的欄位與聚合函式，獲得分組摘要的結果，是一個 `Series`。

```python
daily_report.groupby("Country_Region")['Confirmed'].sum()
```

    Country_Region
    Afghanistan           38324
    Albania               10102
    Algeria               46071
    Andorra                1215
    Angola                 2935
                          ...
    West Bank and Gaza    25575
    Western Sahara           10
    Yemen                  1983
    Zambia                12709
    Zimbabwe               6837
    Name: Confirmed, Length: 188, dtype: int64

## 排序

呼叫 `Series` 的 `sort_values(ascending=False)` 方法將摘要結果由大到小遞減排序。

```python
confirmed_by_country = daily_report.groupby("Country_Region")['Confirmed'].sum()
confirmed_by_country.sort_values(ascending=False)[:10] # 顯示確診人數前 10 高的國家
```

    Country_Region
    US              6244970
    Brazil          4123000
    India           4113811
    Russia          1017131
    Peru             676848
    Colombia         650055
    South Africa     636884
    Mexico           629409
    Spain            498989
    Argentina        471806
    Name: Confirmed, dtype: int64


##### Copyright 2018 The TensorFlow Probability Authors.

Licensed under the Apache License, Version 2.0 (the "License");

```
#@title Licensed under the Apache License, Version 2.0 (the "License"); { display-mode: "form" }
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Fitting Generalized Linear Mixed-effects Models Using Variational Inference

    View on TensorFlow.org

    Run in Google Colab

    View source on GitHub

    Download notebook

## Abstract

In this colab we demonstrate how to fit a Generalized Linear Mixed-effects Model using Variational Inference and TensorFlow.

## Model Family

[Generalized linear mixed-effect models](https://en.wikipedia.org/wiki/Generalized_linear_mixed_model) (GLMM) are similar to [generalized linear models](https://en.wikipedia.org/wiki/Generalized_linear_model) (GLM) except that they incorporate a sample specific noise into the predicted linear response.  This is useful in part because it allows rarely seen features to share information with more commonly seen features.

As a generative process, a Generalized Linear Mixed-effects Model (GLMM) is characterized by:

$$
\begin{align}
\text{for } & r = 1\ldots R:  \hspace{2.45cm}\text{# for each random-effect group}\\
 &\begin{aligned}
  \text{for } &c = 1\ldots |C_r|:  \hspace{1.3cm}\text{# for each category ("level") of group $r$}\\
  &\begin{aligned}
    \beta_{rc}
    &\sim \text{MultivariateNormal}(\text{loc}=0_{D_r}, \text{scale}=\Sigma_r^{1/2})
  \end{aligned}
\end{aligned}\\\\
\text{for } & i = 1 \ldots N:  \hspace{2.45cm}\text{# for each sample}\\
&\begin{aligned}
  &\eta_i = \underbrace{\vphantom{\sum_{r=1}^R}x_i^\top\omega}_\text{fixed-effects} + \underbrace{\sum_{r=1}^R z_{r,i}^\top \beta_{r,C_r(i) }}_\text{random-effects} \\
  &Y_i|x_i,\omega,\{z_{r,i} , \beta_r\}_{r=1}^R \sim \text{Distribution}(\text{mean}= g^{-1}(\eta_i))
\end{aligned}
\end{align}
$$

where:

$$
\begin{align}
R &= \text{number of random-effect groups}\\
|C_r| &= \text{number of categories for group $r$}\\
N &= \text{number of training samples}\\
x_i,\omega &\in \mathbb{R}^{D_0}\\
D_0 &= \text{number of fixed-effects}\\
C_r(i) &= \text{category (under group $r$) of the $i$th sample}\\
z_{r,i} &\in \mathbb{R}^{D_r}\\
D_r &= \text{number of random-effects associated with group $r$}\\
\Sigma_{r} &\in \{S\in\mathbb{R}^{D_r \times D_r} : S \succ 0 \}\\
\eta_i\mapsto g^{-1}(\eta_i) &= \mu_i, \text{inverse link function}\\
\text{Distribution} &=\text{some distribution parameterizable solely by its mean}
\end{align}
$$

In words, this says that every category of each group is associated with an iid MVN, $\beta_{rc}$. Although the $\beta_{rc}$ draws are always independent, they are only indentically distributed for a group $r$; notice there is exactly one $\Sigma_r$ for each $r\in\{1,\ldots,R\}$.

When affinely combined with a sample's group's features ($z_{r,i}$), the result is sample-specific noise on the $i$-th predicted linear response (which is otherwise $x_i^\top\omega$).

When we estimate $\{\Sigma_r:r\in\{1,\ldots,R\}\}$ we're essentially estimating the amount of noise a random-effect group carries which would otherwise drown out the signal present in $x_i^\top\omega$.

There are a variety of options for the $\text{Distribution}$ and [inverse link function](https://en.wikipedia.org/wiki/Generalized_linear_model#Link_function), $g^{-1}$. Common choices are:
- $Y_i\sim\text{Normal}(\text{mean}=\eta_i, \text{scale}=\sigma)$,
- $Y_i\sim\text{Binomial}(\text{mean}=n_i \cdot \text{sigmoid}(\eta_i), \text{total_count}=n_i)$, and,
- $Y_i\sim\text{Poisson}(\text{mean}=\exp(\eta_i))$.

For more possibilities, see the [`tfp.glm`](https://github.com/tensorflow/probability/tree/master/tensorflow_probability/python/glm) module.

## Variational Inference

Unfortunately, finding the maximum likelihood estimates of the parameters $\beta,\{\Sigma_r\}_r^R$ entails a non-analytical integral. To circumvent this problem, we instead find the parameters which minimize an upper bound.  Writing $\phi(x)=\exp(-x^2/2)/\sqrt{2\pi}$ we find:

$\begin{align}
-\log \overbrace{p(\{y\}_i^N|\{x_i,z_i\}_i^N,\beta,\{\Sigma_r\}_r^R)}^{\text{evidence}}
& = -\log \int_{\mathbb{R}^{\sum_r |C_r|D_r}} \overbrace{\left(\prod_i^N p(y_i | \eta_i(u)) \right)}^{\text{likelihood}} \overbrace{\left(\prod_r^R |\Sigma_r^{-1/2}| \prod_c^{|C_r|} \phi(\Sigma_r^{-1/2} u_{rc}) \right)}^{\text{prior}} \, du\\
& = -\log \int_{\mathbb{R}^{\sum_r |C_r|D_r}} \frac{
  q_\lambda(u|\{x_i,y_i,z_i\}_i^N,\beta,\{\Sigma_r\}_r^R)
 }{
  q_\lambda(u|\{x_i,y_i,z_i\}_i^N,\beta,\{\Sigma_r\}_r^R)} \left( \prod_i^N p(y_i | \eta_i(u)) \right) \left(\prod_r^R |\Sigma_r^{-1/2}| \prod_c^{|C_r|} \phi(\Sigma_r^{-1/2} u_{rc}) \right) \, du\\
& \le \text{E}_{q_\lambda(U|\{x_i,y_i,z_i\}_i^N,\beta,\{\Sigma_r\}_r^R)} \left[ -\log
\frac{
  \left( \prod_i^N p(y_i | \eta_i(U)) \right) \left(\prod_r^R |\Sigma_r^{-1/2}| \prod_c^{|C_r|} \phi(\Sigma_r^{-1/2} U_{rc}) \right)
  }{
    q_\lambda(U|\{x_i,y_i,z_i\}_i^N,\beta,\{\Sigma_r\}_r^R)
  }  \right]\\
&= \text{KL}\left[q_\lambda(U|\{x_i,y_i,z_i\}_i^N,\beta,\{\Sigma_r\}_r^R) \Bigg| \left( \prod_i^N p(y_i | \eta_i(U)) \right) \left(\prod_r^R |\Sigma_r^{-1/2}| \prod_c^{|C_r|} \phi(\Sigma_r^{-1/2} U_{rc}) \right) \right]
\end{align}$

The inequality follows from [Jensen's Inequality](https://en.wikipedia.org/wiki/Jensen%27s_inequality).

So, instead of solving:

$$\begin{align}
\{\beta^*, \{\Sigma_r^*\}_r^R\} = \operatorname{\arg\min}_{\beta,\{\Sigma_r\}_r^R} \left\{
 -\log  p(\{y\}_i^N|\{x_i,z_i\}_i^N,\beta,\{\Sigma_r\}_r^R)
\right\}
\end{align}$$

we solve:

$$\begin{align}
\{\lambda^*, \beta^*, \{\Sigma_r^*\}_r^R\} = \operatorname{\arg\min}_{\lambda,\beta,\{\Sigma_r\}_r^R} \left\{
\text{KL}\left[q_\lambda(U|\{x_i,y_i,z_i\}_i^N,\beta,\{\Sigma_r\}_r^R) \Bigg| \left( \prod_i^N p(y_i | \eta_i(U)) \right) \left(\prod_r^R |\Sigma_r^{-1/2}| \prod_c^{|C_r|} \phi(\Sigma_r^{-1/2} U_{rc}) \right) \right]
\right\}
\end{align}$$

## Toy Problem

[Gelman et al.'s (2007) "radon dataset"](http://www.stat.columbia.edu/~gelman/arm/) is a dataset sometimes used to demonstrate approaches for regression. (E.g., this closely related [PyMC3 blog post](http://twiecki.github.io/blog/2014/03/17/bayesian-glms-3/).) The radon dataset contains indoor measurements of Radon taken throughout the United States. [Radon](https://en.wikipedia.org/wiki/Radon) is naturally ocurring radioactive gas which is [toxic](http://www.radon.com/radon_facts/) in high concentrations.

For our demonstration, let's suppose we're interested in validating the hypothesis that Radon levels are higher in households containing a basement. We also suspect Radon concentration is related to soil-type, i.e., geography matters.

To frame this as an ML problem, we'll try to predict log-radon levels based on a linear function of the floor on which the reading was taken.  We'll also use the county as a random-effect and in so doing account for variances due to geography. In other words, we'll use a [generalized linear mixed-effect model](https://en.wikipedia.org/wiki/Generalized_linear_mixed_model).

```
%matplotlib inline

from pprint import pprint
import collections
import matplotlib.pyplot as plt; plt.style.use('ggplot')
import numpy as np
import pandas as pd
import seaborn as sns; sns.set_context('notebook')
import tensorflow.compat.v1 as tf
import warnings
```

### Obtain Dataset:

```
from six.moves import urllib
CACHE_DIR = os.path.join(os.sep, 'tmp', 'radon')

def cache_or_download_file(cache_dir, url_base, filename):
  """Read a cached file or download it."""
  filepath = os.path.join(cache_dir, filename)
  if tf.gfile.Exists(filepath):
    return filepath
  if not tf.gfile.Exists(cache_dir):
    tf.gfile.MakeDirs(cache_dir)
  url = os.path.join(url_base, filename)
  print("Downloading {url} to {filepath}.".format(url=url, filepath=filepath))
  urllib.request.urlretrieve(url, filepath)
  return filepath

def download_radon_dataset(cache_dir=CACHE_DIR):
  """Download the radon dataset and read as Pandas dataframe."""
  url_base = 'http://www.stat.columbia.edu/~gelman/arm/examples/radon/'
  # Alternative source:
  # url_base = ('https://raw.githubusercontent.com/pymc-devs/uq_chapter/'
  #             'master/reference/data/')
  srrs2 = pd.read_csv(cache_or_download_file(cache_dir, url_base, 'srrs2.dat'))
  srrs2.rename(columns=str.strip, inplace=True)
  cty = pd.read_csv(cache_or_download_file(cache_dir, url_base, 'cty.dat'))
  cty.rename(columns=str.strip, inplace=True)
  return srrs2, cty

def preprocess_radon_dataset(srrs2, cty, state='MN'):
  """Preprocess radon dataset as done in "Bayesian Data Analysis" book."""
  srrs2 = srrs2[srrs2.state==state].copy()
  cty = cty[cty.st==state].copy()

  # We will now join datasets on Federal Information Processing Standards
  # (FIPS) id, ie, codes that link geographic units, counties and county
  # equivalents. http://jeffgill.org/Teaching/rpqm_9.pdf
  srrs2['fips'] = 1000 * srrs2.stfips + srrs2.cntyfips
  cty['fips'] = 1000 * cty.stfips + cty.ctfips

  df = srrs2.merge(cty[['fips', 'Uppm']], on='fips')
  df = df.drop_duplicates(subset='idnum')
  df = df.rename(index=str, columns={'Uppm': 'uranium_ppm'})

  df['radon'] = df.activity.apply(lambda x: x if x > 0. else 0.1)

  # Remap categories to start from 0 and end at max(category).
  county_name = sorted(df.county.unique())
  df['county'] = df.county.astype(
      pd.api.types.CategoricalDtype(categories=county_name)).cat.codes
  county_name = map(str.strip, county_name)

  df['log_radon'] = df['radon'].apply(np.log)
  df['log_uranium_ppm'] = df['uranium_ppm'].apply(np.log)
  df = df[['log_radon', 'floor', 'county', 'log_uranium_ppm']]

  return df, county_name
```

```
df, counties = preprocess_radon_dataset(*download_radon_dataset())
```

```
df.head()
```

      log_radon
      floor
      county
      log_uranium_ppm

      0
      0.788457
      1
      0
      -0.689048

      1
      0.788457
      0
      0
      -0.689048

      2
      1.064711
      0
      0
      -0.689048

      3
      0.000000
      0
      0
      -0.689048

      4
      1.131402
      0
      1
      -0.847313

### Specializing the GLMM Family

In this section, we specialize the GLMM family to the task of predicting radon levels. To do this, we first consider the fixed-effect special case of a GLMM.

Using R notation, we might consider the following GLM:

    log(radon) ~ 1 + floor

This model posits that the `radon` response is governed by the floor of a building (e.g., "ground floor" of a "two story" home).  More concretely it states that the `log(radon)` reading is explainable (in expectation) by the formula `offset + weight_floor[floor]`, i.e., there's a weight learned for every floor and a universal `intercept` term. In R's "tilde notation", the `1` indicates there's a weight associated with the "null feature", i.e., a weight associated with every sample.

Given our data, seems like it might be a good start. I.e.,

```
df['log_radon'].plot(kind='density');
plt.xlabel('log(radon)')
plt.figure()
df['floor'].value_counts().plot(kind='bar')
plt.xlabel('Floor')
plt.ylabel('Count')
```

Although this seems like a good start, including something about geography is probably even better. I.e.,

    radon ~ 1 + floor + county

This model posits that the `radon` response is goverened by `offset + weight_floor[floor] + weight_county[county]`, i.e., the same as before except with a county-specific weight.

In the absence of a training set, `radon ~ 1 + floor + county` feels right.  However, if we studied the training data, we'd discover that there's there's a large number of counties with a very small number of measurements. I.e.,

```
fig, ax = plt.subplots(figsize=(22, 5));
county_freq = df['county'].value_counts()
county_freq.plot(kind='bar');
plt.xlabel('County Code');
plt.ylabel('Count');
```

This is worrisome.  If we attempted to fit this model, the `weight_county` vector would likely end up memorizing the results for counties which had only a few training samples. Not good.

GLMM's offer a happy middle to the above two GLMs.  I.e., we might consider fitting,

    radon ~ 1 + floor + (1 | county)

This model is exactly the same as the first, except we've introduced the random-effect, `(1 | county)`.  Adding this random effect has the effect of allowing per-county random fluctuations in radon  In words, `(1 | county)` means that across samples and within a county, the random fluctuation in observed radon will be the same draw from a random Normal . Furthermore, since the covariance is shared among a group (i.e., a "random effect"), the counties with more observations provide a hint at the variance of counties with few observations (since the variance is estimated from the whole group).

## Experiment

We'll now try to fit the `radon ~ 1 + floor + (1 | county)` GLMM using variational inference in TensorFlow. Our only remaining trick is to use stochastic gradient descent. (For brevity, we omit additional details.)

### Imports

```
import tensorflow_probability as tfp

tfd = tfp.distributions
tfb = tfp.bijectors
```

The following code allows `Session` customization.  Feel free to play with the `session_options` arguments and see how computational performance changes.

```
def session_options(enable_gpu_ram_resizing=True,
                    enable_xla=False):
  """Convenience function which sets common `tf.Session` options."""
  config = tf.ConfigProto()
  config.log_device_placement = True
  if enable_gpu_ram_resizing:
    # `allow_growth=True` makes it possible to connect multiple colabs to your
    # GPU. Otherwise the colab malloc's all GPU ram.
    config.gpu_options.allow_growth = True
  if enable_xla:
    # Enable on XLA. https://www.tensorflow.org/performance/xla/.
    config.graph_options.optimizer_options.global_jit_level = (
        tf.OptimizerOptions.ON_1)
  return config

def reset_sess(config=None):
  """Convenience function to create the TF graph and session, or reset them."""
  if config is None:
    config = session_options()
  tf.reset_default_graph()
  global sess
  try:
    sess.close()
  except:
    pass
  sess = tf.InteractiveSession(config=config)

reset_sess()
```

### Setup

For transparency, we'll record all knob settings here.

```
hparams = tf.contrib.training.HParams(
    train_test_split = 0.8,
    train_batch_size = 100,
    test_batch_size = 10,

    dtype=np.float32,
    init_raw_scale=np.log(np.expm1(1.)),  # approx= 0.5413
    scale_diag_offset=1e-3,

    surrogate_posterior_rank=1,

    num_monte_carlo_draws=2,

    train_iterations = int(3e3),

    learning_rate_start = 1e-2,
    learning_rate_num_epochs_per_decay = 10,
    learning_rate_decay_factor = 0.99,
)
```

### Data Munging

We'll use `tf.data.Dataset` to feed the data into the TensorFlow graph. For a nice tutorial on different data loading patterns, see ["How to use Dataset in TensorFlow"](https://towardsdatascience.com/how-to-use-dataset-in-tensorflow-c758ef9e4428).

```
def dataset(df, batch_size):
  with tf.compat.v1.name_scope(name='dataset'):
    feature_cols=['county', 'floor']
    label_cols=['log_radon']
    dataset = tf.data.Dataset.from_tensor_slices((
        df[feature_cols].values.astype(np.int32),
        df[label_cols].values.astype(np.float32),
    )).repeat().batch(batch_size)
    iter_ = tf.data.Iterator.from_structure(
        dataset.output_types, dataset.output_shapes)
    init_op = iter_.make_initializer(dataset)
    features, labels = iter_.get_next()
    features.set_shape([batch_size, 2])
    labels.set_shape([batch_size, 1])
    return init_op, features, labels

DataStats = collections.namedtuple(
    'DataStats',
    [
        'num_train',
        'num_test',
        'num_unique_county',
        'num_unique_floor',
    ])

def split_dataset(df, train_test_split, train_batch_size, test_batch_size):
  """Creates train/test split data."""
  with tf.compat.v1.name_scope(name='split_dataset'):
    train_df = df.sample(frac=train_test_split, random_state=42)
    test_df = df.drop(train_df.index)
    data_stats = DataStats(
        num_train=len(train_df),
        num_test=len(test_df),
        num_unique_county=1 + df['county'].max(),
        num_unique_floor=1 + df['floor'].max(),
    )
    train_init_op, train_features, train_labels = dataset(
        train_df,
        train_batch_size)
    test_init_op, test_features, test_labels = dataset(
        test_df,
        test_batch_size)
    return [
        tf.group([train_init_op, test_init_op]),  # dataset_init
        train_features,
        train_labels,
        test_features,
        test_labels,
        data_stats,
    ]
```

### Specify Model

```
def make_prior():
  def _fn():
    dims = data_stats.num_unique_county
    prior_raw_scale = tf.get_variable(
        name='prior_raw_scale',
        initializer=np.array(hparams.init_raw_scale, hparams.dtype))
    scale = tf.nn.softplus(prior_raw_scale) + hparams.scale_diag_offset
    return tfd.Independent(
        tfd.Normal(loc=np.zeros(dims, hparams.dtype), scale=scale),
        reinterpreted_batch_ndims=1,
        name='prior')
  return tf.make_template('make_prior', _fn)()

def make_likelihood(x, u):
  def _fn():
    intercept = tf.get_variable(
      name='intercept',
      initializer=np.zeros(1, hparams.dtype))
    weights_floor = tf.pad(
        tf.get_variable(
            name='weights_floor',
            initializer=np.zeros(data_stats.num_unique_floor - 1,
                                 hparams.dtype)),
        paddings=[[1, 0]])
    random_effect = tf.transpose(tf.gather(tf.transpose(u), x[:, 0]))
    fixed_effect = intercept + tf.gather(weights_floor, x[:, 1])
    predicted_linear_response = fixed_effect + random_effect
    likelihood_raw_scale = tf.get_variable(
        name='likelihood_raw_scale',
        initializer=np.array(hparams.init_raw_scale, hparams.dtype))
    scale = tf.nn.softplus(likelihood_raw_scale) + hparams.scale_diag_offset
    return tfd.Independent(
        tfd.Normal(loc=predicted_linear_response[..., tf.newaxis],
                   scale=scale),
        reinterpreted_batch_ndims=2,
        name='likelihood')

  return tf.make_template('make_likelihood', _fn)()

def make_surrogate_posterior():
  dims = data_stats.num_unique_county
  def _tril():
    loc = tf.get_variable(
        name='surrogate_loc',
        shape=[dims],
        initializer=tf.zeros_initializer())
    raw_scale_tril = tf.get_variable(
        name='surrogate_raw_scale_tril',
        initializer=np.zeros(dims * (dims + 1) // 2, hparams.dtype))
    scale_tril = tfp.math.fill_triangular(raw_scale_tril)
    new_diag = hparams.scale_diag_offset + tf.nn.softplus(
        hparams.init_raw_scale + tf.diag_part(scale_tril))
    scale_tril = tf.linalg.set_diag(scale_tril, new_diag)
    return tfd.MultivariateNormalTriL(
        loc=loc,
        scale_tril=scale_tril,
        name='surrogate_posterior')
  def _lowrank():
    rank = 1
    loc = tf.get_variable(
        name='surrogate_loc',
        shape=[dims],
        initializer=tf.zeros_initializer())
    raw_scale = tf.get_variable(
        name='surrogate_raw_scale',
        initializer=np.zeros(dims * (1 + rank), hparams.dtype))
    return tfd.MultivariateNormalDiagPlusLowRank(
        loc=loc,
        scale_diag=tf.nn.softplus(hparams.init_raw_scale + raw_scale[:dims]),
        scale_perturb_factor=tf.reshape(raw_scale[dims:], [dims, rank]),
        name='surrogate_posterior')
  return tf.make_template(
      'make_surrogate_posterior',
      _tril if hparams.surrogate_posterior_rank >= dims else _lowrank)()
```

### Main

```
reset_sess()

[
  data_init,
  train_features,
  train_labels,
  test_features,
  test_labels,
  data_stats,
] = split_dataset(
    df,
    hparams.train_test_split,
    hparams.train_batch_size,
    hparams.test_batch_size,
)

prior = make_prior()
print(prior)

surrogate_posterior = make_surrogate_posterior()
print(surrogate_posterior)

minibatch_correction_factor = data_stats.num_train / hparams.train_batch_size

def unnormalized_approx_posterior_log_prob(u):
  likelihood = make_likelihood(train_features, u)
  print(likelihood)
  return (likelihood.log_prob(train_labels) * minibatch_correction_factor
          + prior.log_prob(u))

elbo_loss = tfp.vi.monte_carlo_variational_loss(
    p_log_prob=unnormalized_approx_posterior_log_prob,
    q=surrogate_posterior,
    discrepancy_fn=tfp.vi.kl_reverse,  # same as: Evidence Lower BOund
    num_draws=hparams.num_monte_carlo_draws,
    name='elbo_loss')
print(elbo_loss)

global_step = tf.train.get_or_create_global_step()
learning_rate = tf.train.exponential_decay(
    learning_rate=hparams.learning_rate_start,
    global_step=global_step,
    decay_steps=minibatch_correction_factor * hparams.learning_rate_num_epochs_per_decay,
    decay_rate=hparams.learning_rate_decay_factor,
    staircase=True)
opt = tf.train.AdamOptimizer(learning_rate=learning_rate)
train = opt.minimize(elbo_loss, global_step=global_step)

pprint(tf.trainable_variables())

var_init = tf.global_variables_initializer()
```

    tfp.distributions.Independent("make_prior/prior/", batch_shape=(), event_shape=(85,), dtype=float32)
    tfp.distributions.MultivariateNormalDiagPlusLowRank("make_surrogate_posterior/surrogate_posterior/", batch_shape=(), event_shape=(85,), dtype=float32)
    tfp.distributions.Independent("elbo_loss/expectation/make_likelihood/likelihood/", batch_shape=(2,), event_shape=(100, 1), dtype=float32)
    Tensor("elbo_loss/expectation/Mean:0", shape=(), dtype=float32)
    [<tf.Variable 'make_prior/prior_raw_scale:0' shape=() dtype=float32_ref>,
     <tf.Variable 'make_surrogate_posterior/surrogate_loc:0' shape=(85,) dtype=float32_ref>,
     <tf.Variable 'make_surrogate_posterior/surrogate_raw_scale:0' shape=(170,) dtype=float32_ref>,
     <tf.Variable 'make_likelihood/intercept:0' shape=(1,) dtype=float32_ref>,
     <tf.Variable 'make_likelihood/weights_floor:0' shape=(1,) dtype=float32_ref>,
     <tf.Variable 'make_likelihood/likelihood_raw_scale:0' shape=() dtype=float32_ref>]

```
loss_ = np.zeros(hparams.train_iterations)

sess.run([var_init, data_init])

for iter_ in range(hparams.train_iterations):
  [
      _,
      train_features_,
      train_labels_,
      test_features_,
      test_labels_,
      loss_[iter_],
  ] = sess.run([
      train,
      train_features,
      train_labels,
      test_features,
      test_labels,
      elbo_loss,
  ])
  if iter_ % 200 == 0 or iter_ == hparams.train_iterations - 1:
    print("iter:{:>4}  loss:{:.3f}".format(
        iter_, loss_[iter_]))
```

    iter:   0  loss:1966.783
    iter: 200  loss:1020.058
    iter: 400  loss:833.788
    iter: 600  loss:994.871
    iter: 800  loss:883.134
    iter:1000  loss:892.679
    iter:1200  loss:846.151
    iter:1400  loss:857.963
    iter:1600  loss:902.009
    iter:1800  loss:893.753
    iter:2000  loss:814.118
    iter:2200  loss:841.195
    iter:2400  loss:906.226
    iter:2600  loss:881.371
    iter:2800  loss:839.190
    iter:2999  loss:832.338

### Results

```
surrogate_posterior_cov = surrogate_posterior.covariance()
surrogate_posterior_scale = tf.cholesky(surrogate_posterior_cov )

with tf.variable_scope('make_likelihood', reuse=True):
  intercept = tf.get_variable(name='intercept')
  weights_floor = tf.get_variable(name='weights_floor')

[
    surrogate_posterior_scale_,
    surrogate_posterior_cov_,
    intercept_,
    weights_floor_,
    prior_scale_,
] = sess.run([
    surrogate_posterior_scale,
    surrogate_posterior_cov,
    intercept,
    weights_floor,
    prior.distribution.scale,
])

print('    intercept: ', intercept_)
print('weights_floor: ', weights_floor_)
print('  prior_scale: ', prior_scale_)
```

        intercept:  [ 1.46341455]
    weights_floor:  [-0.71115714]
      prior_scale:  0.333418

We will now plot the training-set loss as a function of training iteration. Note that although the `learning_rate` decays to `0`, there's still noise in the loss owing to the learned parameters being evaluated over different mini-batches.

```
plt.plot(loss_, 'b-');
plt.xlabel('iter');
plt.ylabel('loss');
```

We now plot a nonparametric estimate of the density of the diagonal of the surrogate posterior covariance. Roughly speaking, this shows us how much intrinsic variability there is in each county's `log(radon)` readings.

```
surrogate_posterior_diag_scale_ = np.diag(surrogate_posterior_scale_)
with warnings.catch_warnings():
  warnings.simplefilter("ignore")
  sns.kdeplot(surrogate_posterior_diag_scale_, shade=True, color="r");
  plt.xlabel('posterior std. deviation');
  plt.ylabel('density');
```

We now show a heat map of the covariance matrix. This shows us that just looking at the diagonal was a perfectly reasonable thing to do; the matrix is clearly [diagonally dominant](https://en.wikipedia.org/wiki/Diagonally_dominant_matrix).

```
sns.heatmap(surrogate_posterior_cov_)
plt.title('Surrogate Posterior Covariance')
```

We now conjecture that the county-wise standard deviations might be log-correlated with the number of observations for that county.

```
cnt = np.histogram(
    df['county'],
    np.arange(data_stats.num_unique_county+1))[0]

y = np.stack([cnt, surrogate_posterior_diag_scale_]).T
y = y[y[:,0].argsort()]  # sort by zero-th col
sns.regplot(x=np.log(y[:,0]), y=y[:, 1], color='g')
plt.xlabel('county log-count');
plt.ylabel('posterior std. deviation');
```

Indeed--we do appear to have learned a log-linear relationship between std. deviation and county frequency. This is neat because we never expressed this idea in the model; it is simply apparently true.

## Comparing to `lme4` in R

```
%%shell
exit  # Trick to make this block not execute.

radon = read.csv('srrs2.dat', header = TRUE)
radon = radon[radon$state=='MN',]
radon$radon = ifelse(radon$activity==0., 0.1, radon$activity)
radon$log_radon = log(radon$radon)

# install.packages('lme4')
library(lme4)
fit <- lmer(log_radon ~ 1 + floor + (1 | county), data=radon)
fit

# Linear mixed model fit by REML ['lmerMod']
# Formula: log_radon ~ 1 + floor + (1 | county)
#    Data: radon
# REML criterion at convergence: 2171.305
# Random effects:
#  Groups   Name        Std.Dev.
#  county   (Intercept) 0.3282
#  Residual             0.7556
# Number of obs: 919, groups:  county, 85
# Fixed Effects:
# (Intercept)        floor
#       1.462       -0.693
```

    <IPython.core.display.Javascript at 0x7f9ae4af75d0>

    <IPython.core.display.Javascript at 0x7f9ae4af70d0>

    <IPython.core.display.Javascript at 0x7f9ae4af7090>

    <IPython.core.display.Javascript at 0x7f9ae7b59850>

    <IPython.core.display.Javascript at 0x7f9ae4af73d0>

    <IPython.core.display.Javascript at 0x7f9ae4af7410>

    <IPython.core.display.Javascript at 0x7f9ae7b59310>

    <IPython.core.display.Javascript at 0x7f9ae7b598d0>

The following table summarizes the results.

```
print(pd.DataFrame(data=dict(intercept=[1.462, intercept_[0]],
                             floor=[-0.693, weights_floor_[0]],
                             scale=[0.3282, prior_scale_]),
                   index=['lme4', 'vi']))
```

             floor  intercept     scale
    lme4 -0.693000   1.462000  0.328200
    vi   -0.711157   1.463415  0.333418

This table indicates the VI results are within ~10% of `lme4`'s.  This is somewhat surprising since:
- `lme4` is based on [Laplace's method](https://www.jstatsoft.org/article/view/v067i01/) (not VI),
- we used mini-batch SGD,
- no effort was made in this colab to actually converge,
- minimal effort was made to tune hyperparameters,
- no effort was taken regularize or preprocess the data (eg, center features, etc.).

## Conclusion

In this colab we described Generalized Linear Mixed-effects Models and showed how to use variational inference to fit them using TensorFlow. Although the toy problem only had a few 100 training samples, the techniques used here are identical to what's needed at scale.


Lambda School Data Science

*Unit 2, Sprint 3, Module 1*

---

# Define ML problems

You will use your portfolio project dataset for all assignments this sprint.

## Assignment

Complete these tasks for your project, and document your decisions.

- [ ] Choose your target. Which column in your tabular dataset will you predict?
- [ ] Is your problem regression or classification?
- [ ] How is your target distributed?
    - Classification: How many classes? Are the classes imbalanced?
    - Regression: Is the target right-skewed? If so, you may want to log transform the target.
- [ ] Choose which observations you will use to train, validate, and test your model.
    - Are some observations outliers? Will you exclude them?
    - Will you do a random split or a time-based split?
- [ ] Choose your evaluation metric(s).
    - Classification: Is your majority class frequency > 50% and < 70% ? If so, you can just use accuracy if you want. Outside that range, accuracy could be misleading. What evaluation metric will you choose, in addition to or instead of accuracy?
- [ ] Begin to clean and explore your data.
- [ ] Begin to choose which features, if any, to exclude. Would some features "leak" future information?

```python
import pandas as pd
adoption_url = 'https://data.austintexas.gov/resource/9t4d-g238.csv?$limit=100000'
adoption = pd.read_csv(adoption_url)
# in order to see all of the columns:
pd.options.display.max_columns = 100
```

# Target

```python
adoption.head()
```

      animal_id
      name
      datetime
      monthyear
      date_of_birth
      outcome_type
      outcome_subtype
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color

      0
      A775354
      Charley
      2019-11-12T13:29:00.000
      2019-11-12T13:29:00.000
      2013-06-28T00:00:00.000
      Adoption
      NaN
      Dog
      Neutered Male
      6 years
      Cocker Spaniel Mix
      Black/White

      1
      A775130
      *Slinky
      2019-11-12T13:17:00.000
      2019-11-12T13:17:00.000
      2016-06-25T00:00:00.000
      Adoption
      Foster
      Dog
      Spayed Female
      3 years
      Pit Bull Mix
      Blue/White

      2
      A808385
      NaN
      2019-11-12T13:16:00.000
      2019-11-12T13:16:00.000
      2018-11-08T00:00:00.000
      Transfer
      Partner
      Dog
      Neutered Male
      1 year
      Pug/Chihuahua Shorthair
      White/Brown

      3
      A799709
      *Deeogee
      2019-11-12T13:15:00.000
      2019-11-12T13:15:00.000
      2018-07-11T00:00:00.000
      Adoption
      Foster
      Dog
      Neutered Male
      1 year
      Beagle Mix
      Brown Brindle

      4
      A713661
      Coco
      2019-11-12T12:46:00.000
      2019-11-12T12:46:00.000
      2013-10-10T00:00:00.000
      Return to Owner
      NaN
      Dog
      Spayed Female
      6 years
      Labrador Retriever Mix
      Black/White

```python
adoption.shape
```

    (100000, 12)

The target in this project will be to predict whether or not an animal will be adopted or not (transferred to another shelter or, sadly, euthanized) so that perhaps animal shelters, though overwhelmed, can give some extra love or use unique methods to get those animals that may not have the best odds forever homes.

```python
adoption['outcome_type'].value_counts(dropna=False)
```

    Adoption           44389
    Transfer           29848
    Return to Owner    17520
    Euthanasia          6300
    Died                 968
    Rto-Adopt            507
    Disposal             387
    Missing               61
    Relocate              17
    NaN                    3
    Name: outcome_type, dtype: int64

```python
# this might be a feature that can create leakage, will come back to it.
adoption['outcome_subtype'].value_counts(dropna=False)
```

    NaN                    54846
    Partner                24931
    Foster                  7953
    Rabies Risk             2791
    SCRP                    2642
    Suffering               2504
    Snr                     2272
    In Kennel                508
    Aggressive               360
    Offsite                  281
    Medical                  254
    In Foster                242
    At Vet                   171
    Behavior                  82
    Enroute                   65
    Underage                  29
    Court/Investigation       21
    In Surgery                19
    Possible Theft            15
    Field                      8
    Barn                       4
    Prc                        1
    Customer S                 1
    Name: outcome_subtype, dtype: int64

# Classification or Regression?

There are 9 classes of outcomes but for this project I'd like to focus on the animals that were adopted or not. There are a large number of animals that were returned to owners but that would just be due to them getting out etc but they do have a home so I will not include those in my project. "Rto-adopt" or return to owner adoption will also be included with "return to owner."

For animals that were adopted I will consider that to be:
-adoption
-rto-adopt (return to owner through adoption)

I will combine the following for not adopted:
-transfer
-euthanasia
-relocate
-missing (animals that went missing from the shelter--still unsuccessful in getting them homes)
"Died" and "disposal" are animals that may have died while at the shelter or were brought in that needed to be properly disposed of so I will not include these either as they may have been very ill when brought in.

# How is the target distributed?
## Are the classes imbalanced?

```python
adoption['outcome_type'].value_counts(normalize=True)
```

    Adoption           0.443903
    Transfer           0.298489
    Return to Owner    0.175205
    Euthanasia         0.063002
    Died               0.009680
    Rto-Adopt          0.005070
    Disposal           0.003870
    Missing            0.000610
    Relocate           0.000170
    Name: outcome_type, dtype: float64

will need to drop the rows where the outcome type are the ones listed above to be excluded:
-Return to owner
-Rto-adopt
-Died
-Disposal

```python
# adoption updated to drop the outcomes we are excluding:
adoption_upd = adoption[~adoption['outcome_type'].isin(['Return to Owner',
                                                        'Rto-Adopt',
                                                        'Died',
                                                        'Disposal'])]
```

```python
print(adoption_upd.shape)
adoption_upd['outcome_type'].value_counts()
```

    (80618, 12)

    Adoption      44389
    Transfer      29848
    Euthanasia     6300
    Missing          61
    Relocate         17
    Name: outcome_type, dtype: int64

```python
# need to redefine classes as binary. Adoption as 'adopted' and
# the rest as 'not adopted'.
def new_status(outcome):
    if outcome == 'Transfer' or outcome == 'Euthanasia' or outcome == 'Missing' or outcome == 'Relocate':
      return 'Not adopted'
    else:
      return 'Adopted'

```

```python
adoption_upd = adoption_upd.copy()
adoption_upd['new_outcome_type'] = adoption_upd['outcome_type'].apply(new_status)
```

```python
adoption_upd['new_outcome_type'].value_counts(normalize=True)
```

    Adopted        0.550646
    Not adopted    0.449354
    Name: new_outcome_type, dtype: float64

The classes now are combined into a binary classification and the classes are not imbalanced.

```python
# drop the original 'Outcome_Type' column:
adoption_upd.drop(columns='outcome_type')
```

      animal_id
      name
      datetime
      monthyear
      date_of_birth
      outcome_subtype
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color
      new_outcome_type

      0
      A775354
      Charley
      2019-11-12T13:29:00.000
      2019-11-12T13:29:00.000
      2013-06-28T00:00:00.000
      NaN
      Dog
      Neutered Male
      6 years
      Cocker Spaniel Mix
      Black/White
      Adopted

      1
      A775130
      *Slinky
      2019-11-12T13:17:00.000
      2019-11-12T13:17:00.000
      2016-06-25T00:00:00.000
      Foster
      Dog
      Spayed Female
      3 years
      Pit Bull Mix
      Blue/White
      Adopted

      2
      A808385
      NaN
      2019-11-12T13:16:00.000
      2019-11-12T13:16:00.000
      2018-11-08T00:00:00.000
      Partner
      Dog
      Neutered Male
      1 year
      Pug/Chihuahua Shorthair
      White/Brown
      Not adopted

      3
      A799709
      *Deeogee
      2019-11-12T13:15:00.000
      2019-11-12T13:15:00.000
      2018-07-11T00:00:00.000
      Foster
      Dog
      Neutered Male
      1 year
      Beagle Mix
      Brown Brindle
      Adopted

      5
      A808367
      Daily
      2019-11-12T12:38:00.000
      2019-11-12T12:38:00.000
      2016-11-07T00:00:00.000
      Partner
      Dog
      Intact Female
      3 years
      Australian Cattle Dog/Labrador Retriever
      Cream
      Not adopted

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      99995
      A679740
      NaN
      2014-05-26T16:55:00.000
      2014-05-26T16:55:00.000
      2014-03-25T00:00:00.000
      Partner
      Dog
      Intact Male
      2 months
      Catahoula Mix
      Brown Brindle/White
      Not adopted

      99996
      A679715
      NaN
      2014-05-26T16:54:00.000
      2014-05-26T16:54:00.000
      2014-03-25T00:00:00.000
      Partner
      Dog
      Intact Female
      2 months
      Catahoula Mix
      Tan/White
      Not adopted

      99997
      A677532
      *Taco
      2014-05-26T16:52:00.000
      2014-05-26T16:52:00.000
      2014-03-15T00:00:00.000
      NaN
      Cat
      Neutered Male
      2 months
      Domestic Shorthair Mix
      White
      Adopted

      99998
      A677530
      *Chimichanga
      2014-05-26T16:51:00.000
      2014-05-26T16:51:00.000
      2014-03-15T00:00:00.000
      NaN
      Cat
      Neutered Male
      2 months
      Domestic Shorthair Mix
      Black
      Adopted

      99999
      A679225
      Phoebe
      2014-05-26T16:40:00.000
      2014-05-26T16:40:00.000
      2011-05-17T00:00:00.000
      NaN
      Dog
      Spayed Female
      3 years
      Maltese/Miniature Poodle
      Apricot
      Adopted

80618 rows × 12 columns

# Choose Observations

As mentioned above, since the focus of my project is predicting if animals that are in need of forever homes will be adopted or not, I have already excluded the following observations from my model:
-Return to Owner
-Rto-Adopt
-Died
-Disposal

There are 3 missing values for the outcome ,so we can try to look at the outcome subtype to see if we can determine what happened to them but the remaining animals have been categorized into 'adopted' or 'not adopted.'

```python
adoption_upd.isnull().sum()
```

    animal_id               0
    name                30058
    datetime                0
    monthyear               0
    date_of_birth           0
    outcome_type            3
    outcome_subtype     36354
    animal_type             0
    sex_upon_outcome        3
    age_upon_outcome       25
    breed                   0
    color                   0
    new_outcome_type        0
    dtype: int64

# How to Split Data:

The description of the data set said that Austin is becoming a more pet-friendly city so there may be more animals going in and out of shelters in the more recent data vs the earlier data. I will therefore split the data based on time with the most recent data being the test set and then create a test and validation set with the remaining data.

test = adoption_upd[(adoption_upd['datetime'].dt.year == 2019)]
val =  adoption_upd[(adoption_upd['datetime'].dt.year == 2018)]
train = adoption_upd[(adoption_upd['datetime'].dt.year < 2018)]

```python
adoption_upd.dtypes
```

    animal_id           object
    name                object
    datetime            object
    monthyear           object
    date_of_birth       object
    outcome_type        object
    outcome_subtype     object
    animal_type         object
    sex_upon_outcome    object
    age_upon_outcome    object
    breed               object
    color               object
    new_outcome_type    object
    dtype: object

```python
adoption_upd['datetime'] = pd.to_datetime(adoption_upd['datetime'], infer_datetime_format=True)
```

```python
adoption_upd['datetime'].dt.year.value_counts()
```

    2015    14792
    2019    14292
    2016    14120
    2017    14058
    2018    13361
    2014     9995
    Name: datetime, dtype: int64

```python
# how big to make test set? 2019:
# 14292 observations
```

# Evaluation Metrics

since the classes aren't imbalanced I can use accuracy but will also explore the precision and recall for this problem.

precision positive: correctly predict all the animals that were adopted.

\begin{align}
precision = \frac{accurately \ predicted \ adopted}{total\ predicted \ adopted}
\end{align}

recall positive: of all the animals that were adopted, how many were we able to identify?
\begin{align}
recall = \frac{accurately \ predicted \ adopted}{actually \ adopted}
\end{align}

# Begin to clean data and feature selection

```python
# lets look at the 3 missing values for the outcome type:
adoption_upd[adoption_upd['outcome_type'].isnull()]
# both the outcome type and outcome subtype are missing. Since only 3 rows,
# will drop these observations from the data.
```

      animal_id
      name
      datetime
      monthyear
      date_of_birth
      outcome_type
      outcome_subtype
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color
      new_outcome_type

      2565
      A803963
      Little Bit
      2019-09-27 17:59:00
      2019-09-27T17:59:00.000
      2017-09-09T00:00:00.000
      NaN
      NaN
      Dog
      Intact Male
      NaN
      Miniature Schnauzer
      Gray/Black
      Adopted

      53725
      A737705
      *Heddy
      2016-11-19 16:35:00
      2016-11-19T16:35:00.000
      2013-11-02T00:00:00.000
      NaN
      NaN
      Dog
      NaN
      NaN
      Labrador Retriever Mix
      Black/White
      Adopted

      94975
      A686025
      NaN
      2014-08-16 08:35:00
      2014-08-16T08:35:00.000
      2013-08-15T00:00:00.000
      NaN
      NaN
      Other
      Unknown
      1 year
      Bat Mix
      Brown
      Adopted

will create a wrangle function to do things at one time,
list:
adoption_upd.dropna(subset = ['outcome_type'])
the name column has a lot of missing values, will change NaN's to unknown
adoption_upd['name].fillna("unknown", inplace = True)

the outcome subtype is missing 36353 values, almost half of all of our data, since we will know all of the outcome types and this may cause leakage into the test set because certain outcomes can be deduced from the outcome subtype, I will drop that entire column.
adoption_upd.drop(columns='outcome_subtype')

```python
# the sex_upon_outcome column has 3 missing values, 1 of which is a row that
# will be dropped because the outcome type is missing, but noticed an 'unknown'
# value so check to see if that's common:
adoption_upd['sex_upon_outcome'].value_counts()
# will one hot encode this column.
```

    Neutered Male    27563
    Spayed Female    26009
    Intact Female    10091
    Intact Male       9376
    Unknown           7576
    Name: sex_upon_outcome, dtype: int64

```python
# age_upon_outcome has 25 missing values but date_of_birth has none, lets
# look at how many times 'unknown' shows up in the data:
adoption_upd.isin(['Unknown']).sum()
# the name already has 18 unknowns, so will stick to changing NaN's to unknown.
```

    animal_id              0
    name                  18
    datetime               0
    monthyear              0
    date_of_birth          0
    outcome_type           0
    outcome_subtype        0
    animal_type            0
    sex_upon_outcome    7576
    age_upon_outcome       0
    breed                  0
    color                  0
    new_outcome_type       0
    dtype: int64

```python
adoption_upd.isin(['Other']).sum()
```

    animal_id              0
    name                   0
    datetime               0
    monthyear              0
    date_of_birth          0
    outcome_type           0
    outcome_subtype        0
    animal_type         4455
    sex_upon_outcome       0
    age_upon_outcome       0
    breed                  0
    color                  0
    new_outcome_type       0
    dtype: int64

```python
# to see what kind of animals come in to the shelter
adoption_upd['animal_type'].value_counts()
# initially was thinking of only using cats and dogs but am curious to see the other types.
```

    Dog          39760
    Cat          35962
    Other         4455
    Bird           432
    Livestock        9
    Name: animal_type, dtype: int64

since there are no missing or unknown values for DOB which still seems strange
especially for stray animals that were found but maybe they approximated. We can
create a column where we subtract 2019 from the born on year to get the age and see
if there are a lot of differences.

```python
adoption_upd['date_of_birth'] = pd.to_datetime(adoption_upd['date_of_birth'], infer_datetime_format=True)
```

```python
# lets see what happens when we subtract 11-2019 from the date of birth
now = pd.Timestamp('now')
adoption_upd['calculated_age']=(now.year - adoption_upd['date_of_birth'].dt.year) - ((now.month - adoption_upd['date_of_birth'].dt.month) < 0)
# lets compare the 'calculated_age' column to the 'age_upon_outcome'
```

    0        6
    1        3
    2        1
    3        1
    5        3
            ..
    99995    5
    99996    5
    99997    5
    99998    5
    99999    8
    Name: calculated_age, Length: 80618, dtype: int64

```python
# how many different breeds are there:
adoption_upd['breed'].value_counts()
# 2042...very high cardinality, may need to focus on cats and dogs afterall.
```

    Domestic Shorthair Mix                 26076
    Pit Bull Mix                            4723
    Labrador Retriever Mix                  4375
    Chihuahua Shorthair Mix                 3977
    Domestic Shorthair                      3537
                                           ...
    West Highland/Patterdale Terr              1
    Alaskan Husky/Australian Shepherd          1
    Dachshund Longhair/Miniature Poodle        1
    Schipperke/Catahoula                       1
    Queensland Heeler/Dachshund                1
    Name: breed, Length: 2042, dtype: int64

```python
# how many colors are there
adoption_upd['color'].value_counts()
#512 different values, high cardinality as well.
```

    Black/White               8567
    Black                     7240
    Brown Tabby               5508
    Brown                     3377
    Brown Tabby/White         2806
                              ...
    Tricolor/Brown Brindle       1
    Agouti/Cream                 1
    Cream/Blue Point             1
    Gray/Blue Merle              1
    Tricolor/Orange              1
    Name: color, Length: 512, dtype: int64

```python

```